Replication methods

A table’s replication method controls how its tap finds records after the initial load. FastSync is a transfer optimisation, not a replication method.

Choose a method

Method	Change boundary	Inserts	Updates	Deletes
`LOG_BASED`	Database change log	Yes	Yes	Yes
`INCREMENTAL`	Replication-key maximum	Yes	Only when key advances	No
`FULL_TABLE`	None; reads all rows	Yes	Yes	By table replacement or merge semantics

Prefer LOG_BASED for mutable database tables when the source supports it and log retention can be operated safely. Use INCREMENTAL for append-oriented data with a stable increasing key. Use FULL_TABLE only when complete rescans are acceptable.

Log-based replication

LOG_BASED reads inserts, updates, and deletes from a database change log. PipelineWise supports it for MariaDB/MySQL, PostgreSQL, and MongoDB connectors. Experimental connector status still applies.

An initial table without a bookmark uses FullSync when the route supports it. The same run_tap invocation then starts Singer for ongoing log consumption. The source must retain change-log data until the target-acknowledged bookmark has advanced beyond it.

Warning

Losing a binlog, logical replication slot, WAL range, or change-stream token can make the saved bookmark unrecoverable. Restore the source log when possible; otherwise perform a deliberate resync.

Key-based incremental replication

INCREMENTAL stores the maximum observed replication_key and selects rows whose key is greater than or equal to that bookmark:

SELECT <selected_columns>
  FROM <schema>.<table>
 WHERE <replication_key> >= <saved_maximum>;

The overlap permits replay of the boundary value; the target primary key must deduplicate or merge repeated rows. Updates whose replication key does not advance and source deletes are not detected. Backfilled or non-monotonic keys can therefore leave permanent gaps.

Full-table replication

FULL_TABLE reads every selected row on every run. Supported native routes use FullSync; other routes use the Singer path. Account for a full source scan, staging space, target replacement/merge work, and a longer recovery window.

Use Resync and repair when a one-off rebuild is required without permanently setting the table to FULL_TABLE.