Scheduling

PipelineWise does not include a scheduler. Any scheduler that can run a command, preserve its exit status, and keep the runtime directory available can trigger a pipeline.

Requirements

The scheduler must:

  • run with the same PIPELINEWISE_HOME and PIPELINEWISE_CONFIG_DIRECTORY used during import;

  • provide source, target, secret, and cloud credentials;

  • prevent concurrent execution of the same tap-target pair;

  • retain standard output, standard error, and the PipelineWise run log;

  • treat a non-zero exit as failure; and

  • allow a run to finish or terminate it through pipelinewise stop_tap.

PipelineWise also uses a per-pipeline PID file and refuses a second local instance. Do not rely on that as a distributed scheduler lock without validating the shared filesystem’s locking semantics.

Example

Run separate pipelines independently:

*/5 * * * * pipelinewise run_tap --tap orders --target snowflake
15 * * * * pipelinewise run_tap --tap ledger --target snowflake

Redirecting output is optional when the scheduler captures it. PipelineWise still writes its connector logs under ~/.pipelinewise.

Cadence and overlap

Choose a cadence longer than normal run duration or configure the scheduler to skip overlap. A backlog that makes every run reach the next scheduled start is a throughput or source/target availability problem; increasing frequency makes it worse.

For LOG_BASED sources, source-log retention must exceed the maximum time between the last target acknowledgement and successful recovery. For data-diff, schedule run_data_diff_checks independently from replication and leave enough window_end lag for the target to settle.

Retries and recovery

Retry the same command with unchanged state after a transient failure. Do not advance bookmarks as a retry mechanism. Limit automatic retry frequency so a credential, schema, or database outage does not create an alert or connection storm.

After repeated failure, stop retries, retain evidence, correct the cause, and verify target contents after the next success. See Troubleshooting.