Command Line Interface

PipelineWise’s command line interface allows for a number of operation types on a pipeline.

usage: pipelinewise [-h]
                    {init,run_tap,discover_tap,status,test_tap_connection,sync_tables,import_config,validate}

Positional Arguments

subcommand:

Possible choices: init, run_tap, discover_tap, status, test_tap_connection, sync_tables, import_config, validate

Sub-commands:

init

Initialise and create a sample project. The project will contain sample YAML configuration for every supported tap and target connects.

Positional Arguments

–name:

name of the project

run_tap

Run a specific pipeline

–target:

Target connector id

–tap:

Tap connector id

–extra_log:

Optional: Copy logging into PipelineWise logger to see tap run logs at runtime in STDOUT

stop_tap

Stop running a specific pipeline

–target:

Target connector id

–tap:

Tap connector id

discover_tap

Run a specific tap in discovery mode. Discovery mode is connecting to the data source and collecting information that is required for running the tap.

–target:

Target connector id

–tap:

Tap connector id

status

Prints a status summary table of every imported pipeline with their tap and target.

test_tap_connection

Test the tap connection. It will connect to the data source that is defined in the tap and will return success if it’s available.

–target:

Target connector id

–tap:

Tap connector id

sync_tables

Sync or resync one or more tables from a specific datasource. It performs an initial sync and resets the table bookmarks to their new location.

–target:

Target connector id

–tap:

Tap connector id

–tables:

Optional: Comma separated list of tables to sync from the data source.

import

Import a project directory into PipelineWise. It will create every JSON file required for the tap and target connectors into ~/.pipelinewise.

–dir:

relative path to the project directory to import

–taps:

Optional: Comma seperated list of tap id’s to create.

validate

Validates a project directory with YAML tap and target files.

–dir:

relative path to the project directory with YAML taps and targets.

Environment variables

PIPELINEWISE_HOME

Configures the directory at which PipelineWise expects to find installed tap and targets.

PIPELINEWISE_CONFIG_DIRECTORY

Overrides the default directory at which PipelineWise expects to find configuration files generated by pipelinewise import.