Metadata columns and deletes
Targets can add _SDC_ columns that describe ingestion time and source-delete
events. Configure the behaviour per tap because pipelines sharing a target can
have different delete requirements.
Columns
Column |
Meaning |
|---|---|
|
Time the tap extracted the record. |
|
Time the record entered a target load batch. |
|
Time a source delete event was received; |
Configuration outcomes
Settings |
Metadata |
Delete behaviour |
|---|---|---|
|
Enabled automatically |
Physically removes a target row after a source delete event. |
|
Enabled |
Retains the row and sets |
Both |
Disabled |
Retains the target row without marking the source delete. |
add_metadata_columns: true
hard_delete: true
hard_delete defaults to true. Only LOG_BASED replication emits
individual source-delete events; incremental replication cannot detect deletes.
Full-table and PartialSync publication semantics can remove rows as part of
replacing or reconciling a selected range.
Deprecated since version 0.79.0: Soft delete (hard_delete: false) is scheduled for removal. New pipelines
should use hard delete and model retention in a controlled downstream layer.
Changing delete mode does not rewrite historical target rows. Plan a resync or explicit target migration when existing data must adopt the new behaviour.