.. _contribution: Contribution ------------ PipelineWise is using the `Singer Specification `_ for moving data from Taps to Targets. Adding new taps and targets that are following the singer specification is relatively simple but requires some preparation to make sure the new component is compatible with the PipelineWise framework. To add new item to the supported :ref:`taps` or :ref:`targets` please follow the steps below. Adding new tap '''''''''''''' 1. Make sure your tap is following the Singer specification by following the following criteria: * Configuration JSON is mandatory and defined with the ``--config`` CLI argument * Catalog JSON is mandatory and defined with the ``--catalog`` or ``--properties`` CLI arguments * State JSON is optional and defined with the ``--state`` CLI argument * The tap implements the `Discovery Mode `_ and generates a singer compatible catalog JSON to ``STDOUT`` .. note:: Most of the available Singer Taps meet these criterias but you should always double check when adding new connectors. 2. Extend the ``get_tap_properties`` function in `tap_properties.py `_. We have expanded upon the singer specification to enable some required elements like common naming conventions for stream ID's names, catalog arguments et cetera. We need to register the connectors behaviour to be compatible with the PipelineWise common :ref:`yaml_configuration` file structure. Properties to define in `tap_properties.py `_: .. hint:: +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+ | **tap_stream_id_pattern** | Stream id pattern in the ``properties.json`` generated by the tap. | +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+ | **tap_stream_name_pattern** | Stream name pattern in the ``SCHEMA``, ``RECORD`` and ``STATE`` singer JSON messages. | +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+ | **tap_catalog_argument** | The catalog CLI argument of the tap. This is usually ``--properties`` for old styled singer taps and ``--catalog`` for new styled | | | singer taps. | +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+ | **default_replication_method** | Replication method to use when not specified in :ref:`yaml_configuration` | +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+ | **default_data_flattening_max_level** | Set to 0 if auto data flattening is not required. Data flattening is usually required for taps that extracting data from APIs in JSON | | | messages. | +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+ | **tap_config_extras** | Anything else that's required in the tap ``config.json`` to run. This can be static or dynamically generated values at runtime. | +----------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+ 3. Add your tap type to the allowed values in `tap.json `_ 4. Add your tap type to `ConnectorType Enum `_ 5. Add your tap to the `singer-connectors `_ directory. The new directory should have only one ``requirements.txt`` file with a reference and version of the tap in PyPI. Some taps are forks of the community versions and customised to PipelineWise. If new fork or project required in PyPI please mention this the PR. The Wise team will create the PyPi package. 6. Document your tap in the :ref:`taps_list` section with some YAML examples. The editable documentation is at `GitHub Taps in RST format `_. 7. Send a Pull Request to the `PipelineWise Github Repository `_. Adding new target ''''''''''''''''' 1. Make sure your target is following the Singer specification and meets the following criteria: * Configuration JSON is mandatory and defined with the ``--config`` CLI argument * State messages printed to standard output. PipelineWise compatible target connectors should send `Singer State Messages `_ in the original format to ``STDOUT``. Other components of PipelineWise will process it at a later stage making it compatible across every target connector. 2. Add your target to `singer-connectors `_ directory. The new directory should have only one ``requirements.txt`` file with a reference and version of the tap in PyPI. Some taps are forks of the community versions and customised to PipelineWise. If new fork or project required in PyPI please mention this the PR. The Wise team will create the PyPi package. 3. Add your target type to the allowed values in `target.json `_ 4. Add your target type to `ConnectorType Enum `_ 5. Document your target in the :ref:`targets_list` section with some YAML examples. The editable documentation is at `GitHub Targets in RST format `_. 6. Send a Pull Request to the `PipelineWise Github Repository `_.