Contribution

PipelineWise is using the Singer Specification for moving data from Taps to Targets. Adding new taps and targets that are following the singer specification is relatively simple but requires some preparation to make sure the new component is compatible with the PipelineWise framework.

To add new item to the supported Taps or Targets please follow the steps below.

Adding new tap

1. Make sure your tap is following the Singer specification by following the following criteria:

   • Configuration JSON is mandatory and defined with the --config CLI argument

   • Catalog JSON is mandatory and defined with the --catalog or --properties CLI arguments

   • State JSON is optional and defined with the --state CLI argument

   • The tap implements the Discovery Mode and generates a singer compatible catalog JSON to STDOUT

Note

Most of the available Singer Taps meet these criterias but you should always double check when adding new connectors.

2. Extend the get_tap_properties function in tap_properties.py.

We have expanded upon the singer specification to enable some required elements like common naming conventions for stream ID’s names, catalog arguments et cetera. We need to register the connectors behaviour to be compatible with the PipelineWise common YAML configuration file structure. Properties to define in tap_properties.py:

Hint

tap_stream_id_pattern	Stream id pattern in the properties.json generated by the tap.
tap_stream_name_pattern	Stream name pattern in the SCHEMA, RECORD and STATE singer JSON messages.
tap_catalog_argument	The catalog CLI argument of the tap. This is usually --properties for old styled singer taps and --catalog for new styled singer taps.
default_replication_method	Replication method to use when not specified in YAML configuration
default_data_flattening_max_level	Set to 0 if auto data flattening is not required. Data flattening is usually required for taps that extracting data from APIs in JSON messages.
tap_config_extras	Anything else that’s required in the tap config.json to run. This can be static or dynamically generated values at runtime.

3. Add your tap type to the allowed values in tap.json

4. Add your tap type to ConnectorType Enum

5. Add your tap to the singer-connectors directory. The new directory should have only one requirements.txt file with a reference and version of the tap in PyPI. Some taps are forks of the community versions and customised to PipelineWise. If new fork or project required in PyPI please mention this the PR. The Wise team will create the PyPi package.

6. Document your tap in the Taps (Data Sources) section with some YAML examples. The editable documentation is at GitHub Taps in RST format.

7. Send a Pull Request to the PipelineWise Github Repository.

Adding new target

1. Make sure your target is following the Singer specification and meets the following criteria:

   • Configuration JSON is mandatory and defined with the --config CLI argument

   • State messages printed to standard output. PipelineWise compatible target connectors should send Singer State Messages in the original format to STDOUT. Other components of PipelineWise will process it at a later stage making it compatible across every target connector.

2. Add your target to singer-connectors directory. The new directory should have only one requirements.txt file with a reference and version of the tap in PyPI. Some taps are forks of the community versions and customised to PipelineWise. If new fork or project required in PyPI please mention this the PR. The Wise team will create the PyPi package.

3. Add your target type to the allowed values in target.json

4. Add your target type to ConnectorType Enum

5. Document your target in the Targets (Destinations) section with some YAML examples. The editable documentation is at GitHub Targets in RST format.

6. Send a Pull Request to the PipelineWise Github Repository.