Skip to content
Optima docs
ConsoleIntegrationsData connectors

Change data capture

Import row changes from a database as they happen by unwrapping change events on a stream source.

Change data capture (CDC) streams every insert, update and delete of a source database table into Optima. Optima reads the change events that Debezium publishes, unwraps them, and treats each as an ordinary row. It is an option on stream sources, set in the New connector or Edit connector dialog of Data connectors.

Needs the integration_editor role and the Connect licence feature, or another role your administrator has given access to this area — see Licences and roles.

Fields

FieldMeaningValid valuesDefault
Debezium change events (CDC)Unwraps Debezium change events before mapping. Snapshot reads are imported as ordinary rows. Field mappings and scripts then work on the table's columns instead of payload.after.On or offOff
Delete eventsWhat happens when the source row is deleted.Skip (default), Emit before-image, Delete the recordSkip

The Delete events choices:

  • Skip drops deletes, truncates and tombstones. They are counted as skipped.
  • Emit before-image forwards the deleted row's last values, marked __op = "d". Your transform script should check for that marker.
  • Delete the record removes the record for real. Available for Events and Custom dataset connectors only. A dataset record is found by its dataset_record_key, an event by its event_key, which your transform script sets or which arrives in the canonical record shape. Profiles cannot be deleted this way: erase them with an erasure request.

The Quick start: Debezium CDC button in the New connector dialog selects a NATS stream source with change events switched on. When the protocol is NATS, JetStream is switched on automatically because plain NATS delivers at most once and a failed change event would be lost.

What your script sees

With CDC on, the change events are unwrapped before the transform script runs. transform(payload) receives the bare table row, with columns as top-level keys. There is no payload.after and no operation code. Tombstones, truncates and skipped deletes never reach the script.

Inserts, updates and snapshot reads therefore look identical. To tell them apart, configure Debezium's ExtractNewRecordState transform with add.fields=op and read payload.get("__op"): "c" is an insert, "u" an update and "r" a snapshot read. With delete events set to emit, the before-image arrives with __op equal to "d". The script tester does not unwrap events, so use bare rows as sample payloads.

def transform(payload):
    if payload.get("__op") == "d":
        return None
    return {
        "identifiers": [{"key": "email", "value": payload["email"]}],
        "attributes": {"plan": payload["plan"]},
    }

Examples

Mirror a customers table. Quick start, then pick the Profiles data type, an Entity type, and map the table's columns on the connector detail page. Leave Delete events on Skip so removing a row in the source never removes the profile.

Keep a dataset in step with a product table. Use Custom dataset, set Delete events to Delete the record, and have the script return dataset_record_key from the primary key.

Common problems

  • "Delete the record" is missing. It exists only for Events and Custom dataset data types.
  • Deletes re-create the row. Emit before-image re-creates the row unless your script handles __op = "d".
  • A delete is counted as failed. A delete event with no usable key (no event_key or dataset_record_key) is counted under Rows failed and is never retried.
  • A delete seems to do nothing. If the delete arrives before the insert it refers to, there is nothing to remove and it is silently ignored.
  • I do not know the broker address. The NATS URL and the credentials for the Quick start come from your administrator.
  • Field mappings do not match. With CDC they target the table's columns, not payload.after.*.

On this page