Create a pipeline

A PipelineWise project contains global configuration plus one YAML file per source and target. import_config validates these files, discovers source schemas, and generates the JSON, catalog, and state files used at runtime.

This walkthrough configures the available PostgreSQL-to-Snowflake route with INCREMENTAL replication. See PostgreSQL source and Snowflake target for production prerequisites.

Generate a project

pipelinewise init --name pipelinewise_samples
cd pipelinewise_samples
mv tap_postgres.yml.sample tap_postgres.yml
mv target_snowflake.yml.sample target_snowflake.yml

pipelinewise init also writes templates for experimental and legacy connectors. A template is not evidence that its connector is packaged or available; check Connectors.

Configure Snowflake

Keep only the required target settings in target_snowflake.yml:

id: "snowflake"
name: "Analytics Snowflake"
type: "target-snowflake"
db_conn:
  account: "<ACCOUNT>"
  dbname: "<DATABASE>"
  user: "<USER>"
  private_key: "/run/secrets/snowflake-key.pem"
  warehouse: "<WAREHOUSE>"
  s3_bucket: "<STAGING_BUCKET>"
  s3_key_prefix: "pipelinewise/"
  stage: "<SCHEMA>.<STAGE>"
  file_format: "<SCHEMA>.<FILE_FORMAT>"

The stage, file format, role, and user must exist before the first run. Use an instance role or AWS profile where possible; avoid committing static credentials.

Configure PostgreSQL

Edit tap_postgres.yml. The source user needs metadata access and SELECT on the configured table. INCREMENTAL requires a stable replication key.

id: "orders"
name: "Orders PostgreSQL"
type: "tap-postgres"
owner: "data-platform@example.com"
db_conn:
  host: "<HOST>"
  port: 5432
  user: "<USER>"
  password: "{{ env_var['POSTGRES_PASSWORD'] }}"
  dbname: "<DATABASE>"
target: "snowflake"
batch_size_rows: 20000
stream_buffer_size: 0
schemas:
  - source_schema: "public"
    target_schema: "repl_orders"
    tables:
      - table_name: "orders"
        replication_method: "INCREMENTAL"
        replication_key: "updated_at"

The target value must equal the target YAML id. For LOG_BASED replication, configure logical decoding before importing; see PostgreSQL source.

Provide secrets

Jinja expressions read environment variables when PipelineWise loads YAML:

password: "{{ env_var['POSTGRES_PASSWORD'] }}"

Alternatively, encrypt values with Ansible Vault as described in Encrypt configuration values. Keep plaintext secrets, vault password files, and private keys outside source control.

Validate and import

Run validation before PipelineWise connects to the source:

pipelinewise validate --dir .
pipelinewise import_config --dir .
pipelinewise status

validate checks YAML structure and references but does not test database connectivity. import_config performs discovery and writes runtime files below ~/.pipelinewise/<target_id>/<tap_id>/. Commit the project YAML, not the generated runtime directory.

Continue with Run a pipeline after the imported pipeline reports ready.