Command-line interface

PipelineWise commands operate on a project directory or an imported tap-target pair. Commands exit non-zero on validation, discovery, replication, or reconciliation failure; automation must retain that exit status and the run log.

Command summary

Command	Purpose	Changes state or data
`init`	Generate a project and connector templates.	Creates local files.
`validate`	Validate project YAML and references.	No.
`import_config`	Discover sources and generate runtime configuration.	Replaces generated config; persists data-diff definitions.
`status`	List imported pipelines and last results.	No.
`test_tap_connection`	Test source connectivity.	No target load.
`discover_tap`	Refresh or inspect source catalog metadata.	Runs source discovery.
`run_tap`	Run normal initial and ongoing replication.	Loads target data and advances acknowledged state.
`stop_tap`	Terminate a managed running pipeline.	Stops processes; may leave replayable work.
`fast_sync`	FullSync or configured PartialSync selected tables.	Rebuilds/merges data and resets bookmarks.
`partial_sync_table`	Repair one bounded source range.	Merges target data; can update state.
`reset_state`	Move CDC state after a controlled switchover.	Changes bookmarks without copying data.
`encrypt_string`	Produce an Ansible Vault YAML value.	No pipeline state.
`list_data_diff_checks`	Inspect persisted data-diff definitions and coverage.	No.
`run_data_diff_checks`	Run due data-diff checks.	Persists attempts and coverage.
`rerun_data_diff_check`	Re-run one failed immutable window.	Persists a remediation attempt.

Project lifecycle

`init`

pipelinewise init --name <project>

Creates sample YAML for available, experimental, and some legacy connectors. Review Connectors before enabling a template.

`validate`

pipelinewise validate --dir <project>

Checks YAML syntax, required fields, connector types, schema mapping, and tap target references. It does not connect to a source or target.

`import_config`

pipelinewise import_config --dir <project>

Useful options:

Option	Behaviour
`--taps <id,id>`	Imports only the named tap IDs and their targets.
`--secret <file>`	Reads the Ansible Vault password needed by encrypted YAML values.

The command validates, connects to sources, performs discovery, and writes generated files below ~/.pipelinewise. Data-diff definitions are versioned only after connector generation and discovery succeed. import remains a deprecated alias.

Warning

Removing or renaming a tap or target in project YAML makes import_config delete its generated runtime directory and saved state. Removing a PostgreSQL tap also drops its replication slot. Back up state and plan a new initial sync before importing that change.

Inspect and test

`status`

pipelinewise status

Shows imported tap-target pairs, enabled state, current status, and last run result. A successful status row does not prove source-to-target equality.

`test_tap_connection`

pipelinewise test_tap_connection --tap <tap_id> --target <target_id>

Tests the configured source connection. It does not test the target or load records.

`discover_tap`

pipelinewise discover_tap --tap <tap_id> --target <target_id>

Runs source discovery and writes catalog metadata. Use it to diagnose missing schemas, tables, fields, or permissions. import_config runs discovery automatically.

Replication

`run_tap`

pipelinewise run_tap --tap <tap_id> --target <target_id>

Runs eligible initial FastSync work, then Singer replication in the same invocation. --extra_log mirrors connector output to the PipelineWise logger. See Run a pipeline for preflight and recovery.

`stop_tap`

pipelinewise stop_tap --tap <tap_id> --target <target_id>

Signals the process tree associated with the tap-target PID file. The target is given an opportunity to finish data already received. Restart without editing state after an unexpected interruption.

`fast_sync`

pipelinewise fast_sync \
  --tap <tap_id> \
  --target <target_id> \
  --tables <schema.table,schema.table>

Important options
Option	Behaviour
`--tables`	Limits work to comma-separated source `schema.table` names.
`--force`	Overrides `allowed_resync_max_size`.
`--replication_method_only`	Selects tables with `full_table`, `incremental`, or `log_based`.

The command fails when FullSync is unavailable for the route. It never falls back to Singer. A table with sync_start_from uses PartialSync. sync_tables remains a deprecated alias.

`partial_sync_table`

pipelinewise partial_sync_table \
  --tap <tap_id> \
  --target <target_id> \
  --table <schema.table> \
  --column <column> \
  --start_value <inclusive_start> \
  --end_value <inclusive_end>

--end_value is optional. When absent, PipelineWise captures the current replication position and can update state after the merge. PartialSync is available only from MariaDB/MySQL or PostgreSQL to Snowflake. See PartialSync behaviour.

`reset_state`

pipelinewise reset_state --tap <tap_id> --target <target_id>

Use this only after a controlled MariaDB/MySQL or PostgreSQL switchover whose old and new replication positions are known. The command changes state without copying rows; an incorrect mapping can skip data permanently.

For MariaDB/MySQL, switch_over_data_file in config.yml points to JSON that maps the new host to the old/new identifiers, hosts, timestamp, and binlog positions. Back up state and verify target continuity after the first run.

Data-diff

`list_data_diff_checks`

pipelinewise list_data_diff_checks --target <target_id> --tap <tap_id>

Options include --output-format table|json and --include-versioned. --tap requires --target.

`run_data_diff_checks`

pipelinewise run_data_diff_checks --target <target_id> --tap <tap_id>
pipelinewise run_data_diff_checks --all

--check selects a check name, logical key, or version ID. --force creates another attempt for the current UTC slot when a terminal attempt already exists.

`rerun_data_diff_check`

pipelinewise rerun_data_diff_check \
  --run-id <uuid> \
  --remediation-ref <ticket_or_incident>

Both options are required. The original attempt remains immutable. See Data-diff checks for scheduling, coverage, and remediation semantics.

Secrets

`encrypt_string`

pipelinewise encrypt_string \
  --secret <vault-password-file> \
  --string <value>

The command prints an Ansible Vault YAML value. Avoid shared shell history and process inspection when supplying sensitive command-line values. See Encrypt configuration values.

Common options and environment

Option	Behaviour
`--log <file>`	Writes PipelineWise CLI logs to a file.
`--extra_log`	Copies Singer and FastSync subprocess output to standard output.
`--debug`	Enables debug logging on standard output.
`--profiler` / `-p`	Writes cProfile output below the configured profiling directory.
`--version`	Prints installed component versions.

PIPELINEWISE_HOME selects the installation root containing connector virtual environments. It defaults to ~/pipelinewise.

PIPELINEWISE_CONFIG_DIRECTORY selects the runtime configuration, state, and log directory. It defaults to ~/.pipelinewise.

Command-line interface

Command summary

Project lifecycle

init

validate

import_config

Inspect and test

status

test_tap_connection

discover_tap

Replication

run_tap

stop_tap

fast_sync

partial_sync_table

reset_state