PipelineWise

Installation

  • Installation
    • Choose an installation method
    • Docker installation
    • Source installation
    • Packaged connectors
  • Create a pipeline
    • Generate a project
    • Configure Snowflake
    • Configure PostgreSQL
    • Provide secrets
    • Validate and import
  • Run a pipeline
    • Preflight
    • Start and observe
    • Stop safely
    • Confirm success
    • Recover from failure

Concepts

  • Singer data flow
    • Messages
    • Taps
    • Targets
    • Acknowledgement boundary
  • Replication methods
    • Choose a method
    • Log-based replication
    • Key-based incremental replication
    • Full-table replication
  • FastSync
    • Supported routes
    • Automatic selection and handover
    • Explicit FullSync
    • Configured PartialSync
  • Stream buffering and acknowledgement
    • Configuration
    • Acknowledgement boundary
    • Termination and restart
    • Operational guidance

Operations

  • YAML configuration
    • Project files
    • Global configuration
    • Tap configuration
    • Schemas and tables
    • Target configuration
    • Secrets and validation
  • Encrypt configuration values
    • Create the vault password file
    • Encrypt a value
    • Import encrypted YAML
    • Rotation and recovery
  • Command-line interface
    • Command summary
    • Project lifecycle
      • init
      • validate
      • import_config
    • Inspect and test
      • status
      • test_tap_connection
      • discover_tap
    • Replication
      • run_tap
      • stop_tap
      • fast_sync
      • partial_sync_table
      • reset_state
    • Data-diff
      • list_data_diff_checks
      • run_data_diff_checks
      • rerun_data_diff_check
    • Secrets
      • encrypt_string
    • Common options and environment
  • Data-diff checks
    • Supported routes
    • Check types
    • Configuration
      • Backend database
      • Tap YAML
      • Choosing a frequency and window
    • CLI commands
    • Alerts
      • One alert per failed check per window
    • Coverage and remediation
      • Monitoring views
      • Source safety
      • Preflight
  • Scheduling
    • Requirements
    • Example
    • Cadence and overlap
    • Retries and recovery
  • Multi-server operation
    • Prerequisites
    • Scheduling boundary
    • Failure recovery
  • Logging
    • Filename fields
    • Observe a run
    • Diagnose failure
    • Retention
  • Alerts
    • Handlers
    • Slack
    • VictorOps
    • Routing and suppression
  • Resync and repair
    • Choose an operation
    • Preflight
    • Full resync
    • Partial repair
    • Failure and validation
  • PartialSync behaviour
    • Range semantics
    • Merge outcomes
    • Visual examples
    • Safety and validation
  • Schema changes
    • Change outcomes
    • Column versioning
    • Operational procedure
  • Load-time transformations
    • Transformation types
    • Configuration
    • Nested JSON fields
    • Conditions
    • Validation and rollout
  • Metadata columns and deletes
    • Columns
    • Configuration outcomes
  • Troubleshooting
    • Symptom index
    • Replication checks
      • When to resync
      • INCREMENTAL method and missing updates
      • Using replica_host for large table resyncs (1 TB+)
      • Changes to transformations
      • Resync table size limit
      • MariaDB / MySQL Errors
        • Lost connection to MySQL server (Errno 104, Connection reset by peer)
        • Unknown encoding: utf8mb3
        • Bogus data in log event
        • Log event exceeded max_allowed_packet
        • Missing binlog
        • Changing the binlog position in state.json
      • PostgreSQL Errors
        • requires wal_level >= logical
        • PGRES_COPY_BOTH and no message from the libpq
        • LOG_BASED replication is slow or falling behind
        • Canceling statement due to conflict with recovery
        • Connection already closed
        • recovery is in progress
        • Unable to find replication slot
      • FastSync Errors
        • Table not found
        • Failure to import tap (0 tables discovered)
      • Logging and Diagnostics
    • Verify recovery

Connectors

  • Connector overview
  • Sources
    • MariaDB and MySQL source
    • PostgreSQL source
    • Snowflake source
    • MongoDB source
    • S3 CSV source
    • Kafka source
    • Zendesk source
    • Jira source
    • GitHub source
    • Slack source
    • Mixpanel source
    • Salesforce source
    • Twilio source
  • Targets
    • PostgreSQL target
    • Snowflake target
    • Snowflake Iceberg tables
    • S3 CSV target

Project

  • Contributing
    • Development environment
    • Add or update a source connector
    • Add or update a target connector
    • Support status
    • Validation
  • Project
    • History
    • Early contributors
    • Contribution
    • Resources and links
  • Licenses
    • Packaged components
PipelineWise
  • Search

© Copyright 2026, Wise Ltd..

Built with Sphinx using a theme provided by Read the Docs.