PipelineWise
PipelineWise is a Python framework for configuring, running, and operating Singer-based ELT pipelines. It moves source data into analytical destinations with log-based, incremental, or full-table replication and optional native bulk transfer through FastSync.
Important
Project scope
Available sources are MariaDB and PostgreSQL; available targets are
PostgreSQL and Snowflake. Other packaged connectors, including the Snowflake
source, are experimental. pipelinewise init also generates some legacy
templates that are not packaged. Review Connectors before
selecting a source and target.
Release v0.64.1 is the last version from before the connector set was
reduced. It is a historical reference, not a recommendation to deploy an
older release.
Start here
Installation — install PipelineWise, preferably with Docker.
Create a pipeline — configure an available source-to-target route.
Run a pipeline — validate, import, run, and inspect a pipeline.
Troubleshooting — diagnose failures and recover safely.
Core capabilities
Capability |
Behaviour |
|---|---|
Singer replication |
Log-based change capture, key-based incremental loads, and full-table snapshots. |
FastSync |
Native FullSync and PartialSync transfers for supported database routes. |
Schema evolution |
Detects source schema changes and applies compatible target changes. |
Load-time protection |
Masks, hashes, or removes sensitive values before target loading. |
Data-diff |
Performs bounded aggregate reconciliation with auditable coverage and remediation. |
Configuration as code |
Generates connector JSON, catalogs, and state from version-controlled YAML. |
Documentation
Installation
Operations
Connectors