Data quality watchdog
A built-in agent that checks every day whether events, Connect runs and identifiers still arrive as expected.
The data quality watchdog is a built-in agent that notices when data stops flowing before your campaigns or reports suffer. Once a day it checks event volume per source, failed Connect runs and how many new profiles carry each identifier. It sends one plain-text email, but only when something is flagged.
Who uses it
Tenant administrators and data teams create the instance. You need the Built-in agents feature in your licence. The Connect check also needs the Connect package; without it the agent still runs, skips that check and says so in the email. The owner of the instance needs clearance to read the identifiers you list and a role that can count events.
Fields
| Parameter | What you enter | Default |
|---|---|---|
recipients | The people who get the email. | Owner (you) |
schedule | When the check runs, as a cron expression in the instance time zone. | 0 7 * * * (every day, 07:00) |
sources | Required. The exact event source values to watch, for example edge for the web gateway, connector:<connector id> for a Connect batch connector or stream:<connector id> for a stream source. An event's Source in the console shows the value. | None |
drop_threshold_pct | How far a figure may fall below its comparison before it is flagged, in percent. Above 0 and up to 100; any other value falls back to 30 and the email says so. | 30 |
identifiers | The identifier keys whose fill rate to check. | email, phone |
entity_type | The entity type whose new profiles are checked. | customer |
purpose | The processing purpose the event counts are read for. A tenant in strict privacy mode must set it. | Empty |
always_send | Send a short "all checks passed" email even when nothing is flagged. | Off |
Actions
- Reads: event counts per source, recent Connect runs and counts of new profiles. It reads aggregates, not individual customers.
- Sends: one plain-text email to the recipients you choose, with the subject "Data quality findings" and the date.
The three checks:
- Event volume per source: yesterday's count against the same 24 hours on each of the previous four weeks. A source is flagged when yesterday had no events but an earlier week had, or when yesterday is clearly below the lowest earlier count. The email states yesterday's count and the four earlier counts, not a percentage.
- Connect runs: every run in the last 24 hours that failed or ended partial, with its connector, start time, status, failed rows and error message.
- Identifier fill rates: for each identifier, how many of the new profiles of the last 7 days have it, against how many of the new profiles of the 7 days before. The email states both pairs of counts.
The email lists "What I could not check": failed reads, sources or identifiers left out and "Connect runs not checked (no Connect licence)" when that check was skipped.
A check that cannot run is a finding, not silence: a failed read, a source with no events in any of the five windows, an identifier the entity type does not declare and an identifier that no new profile has in either week all make the email go out, whatever Always send says. Without any finding no email is sent unless Always send is on. When the schema description is too large to read in full, the email says the declarations were not checked and no identifier is reported as undeclared. Every figure is copied from Optima's own results and checked against what the agent read.
What it does not check
It does not detect schema changes or duplicate profiles; Optima does not record those today.
Limits
One run checks at most four sources and three identifiers, the first ones listed; the email names the rest. For more, create one instance per group of sources.
Examples
To try it, create the instance in dry run with sources set to edge, a cheap model and schedule a few minutes ahead. Open the run log to see what it would send, then switch the instance to live.
Common problems
- No email today. Nothing was flagged and Always send is off, or the instance is stopped.
- A source is flagged as having no events in any window. The value in
sourcesis probably not the exact source of your events; check an event's Source in the console. - The email says a source could not be checked because of the privacy mode. Set
purposeto a purpose the owner may read. - The email says the identifier declarations could not be checked. The list of entity types was too large to read in full; every identifier was still queried.
- An identifier is flagged as not declared. The entity type does not declare it; check
identifiersandentity_type. If it is flagged as missing from every new profile, the owner may have no clearance to read it, or nothing fills it. - The card is missing. Your licence has no Built-in agents feature.