Installation

PipelineWise requires Linux-compatible connector tooling and Python 3.12. Running the published application image is the recommended installation method; it isolates connector dependencies in the same environment used at runtime.

Choose an installation method

Method	Use when	Operational consequence
Docker	The host can run Docker.	Connector and system dependencies are isolated from the host.
Source installation	Docker is unavailable or the image must be customised.	The operator owns Python, system-library, and connector dependency compatibility.

Docker installation

Install Docker Engine or Docker Desktop, then pull the published full image. Pin a release tag instead of latest in production.

git clone https://github.com/transferwise/pipelinewise.git
cd pipelinewise
docker pull transferwiseworkspace/pipelinewise:latest
docker tag transferwiseworkspace/pipelinewise:latest pipelinewise:latest

The local tag is required because bin/pipelinewise-docker runs pipelinewise:latest. Published variants are:

Published images
Image	Contents
`transferwiseworkspace/pipelinewise:<version>`	PipelineWise with every packaged connector.
`transferwiseworkspace/pipelinewise:<version>-default`	PipelineWise with only the default connector subset.
`transferwiseworkspace/pipelinewise-barebone:<version>`	PipelineWise without pre-installed connector environments.

Packaging experimental connectors does not change their support level; see Connectors.

Use the wrapper to mount the current project and ~/.pipelinewise into each short-lived container:

alias pipelinewise="$(pwd)/bin/pipelinewise-docker"
pipelinewise status

A successful empty installation prints 0 pipeline(s). The wrapper persists generated configuration, state, and logs below ~/.pipelinewise on the host.

To customise system packages or connector selection, build a local image instead of pulling one:

docker build -t pipelinewise:latest .

Source installation

Install Python 3.12, venv, gettext/envsubst, and the operating-system libraries required by the selected connectors. Then install the CLI and only the connectors needed by the pipeline:

git clone https://github.com/transferwise/pipelinewise.git
cd pipelinewise
make pipelinewise
make connectors -e pw_connector=tap-postgres,target-snowflake

PipelineWise creates an isolated virtual environment per component below ${PIPELINEWISE_HOME}/.virtualenvs. Activate the CLI environment before use:

export PIPELINEWISE_HOME="$(pwd)"
source .virtualenvs/pipelinewise/bin/activate
pipelinewise status

Do not install the root project with pip as a replacement for the Makefile workflow; that does not create connector environments.

Packaged connectors

make all_connectors and the default Docker build package the following components. Install a subset with a comma-separated pw_connector value.

Sources
Connector	Status	Source
`tap-mysql`	Available / Experimental	MariaDB / MySQL
`tap-postgres`	Available	PostgreSQL
`tap-snowflake`	Experimental	Snowflake
`tap-github`	Experimental	GitHub
`tap-jira`	Experimental	Jira
`tap-kafka`	Experimental	Kafka
`tap-mixpanel`	Experimental	Mixpanel
`tap-mongodb`	Experimental	MongoDB
`tap-s3-csv`	Experimental	S3 CSV
`tap-salesforce`	Experimental	Salesforce
`tap-slack`	Experimental	Slack
`tap-twilio`	Experimental	Twilio
`tap-zendesk`	Experimental	Zendesk

Targets and transformations
Component	Status	Purpose
`target-postgres`	Available	Load Singer records into PostgreSQL.
`target-snowflake`	Available	Load Singer records into Snowflake.
`target-s3-csv`	Experimental	Write Singer records as CSV files in S3.
`transform-field`	Available	Apply configured load-time transformations.

Warning

Connector licenses can differ from the PipelineWise license. Review Licenses before distributing an image or installing a connector.

Continue with Create a pipeline after pipelinewise status succeeds.