Singer data flow

PipelineWise uses the Singer specification as the record protocol between source taps and targets:

tap | optional transform-field | optional mbuffer | target

FastSync bypasses this protocol for supported native bulk-transfer routes. See FastSync.

Messages

Message	Producer	Purpose
SCHEMA	Tap	Describes stream fields, types, keys, and schema changes.
RECORD	Tap	Carries one source record or delete event.
STATE	Tap and target	Carries replication bookmarks and target acknowledgement.

Taps

A tap reads connector configuration, a selected catalog, and optional prior state. It writes ordered Singer messages to standard output. PipelineWise owns the generated JSON files; operators configure the source through YAML and should not edit generated catalog or state while a tap is running.

See Taps (Data Sources) for source status and connector-specific requirements.

Targets

A target consumes Singer messages, creates or evolves destination tables, loads records, and emits acknowledged state. PipelineWise persists target-emitted state only after the target process has accepted the corresponding data.

Singer interoperability does not guarantee that every packaged tap-target pair has production support. Use Connectors for endpoint and route status.

Acknowledgement boundary

The source’s latest consumed position can be ahead of the state durably accepted by the target. PipelineWise treats the target-emitted state as the recovery boundary. Buffered or in-memory records beyond that boundary must remain replayable after interruption.

For PostgreSQL LOG_BASED replication, the logical slot’s flush feedback is bounded by the minimum acknowledged LSN in state.json. See Stream buffering and acknowledgement for termination and replay behaviour.