Skip to content
Optima docs
ConsoleAgentsBuilt-in agents

Data quality watchdog

A built-in agent that checks every day whether events, Connect runs and identifiers still arrive as expected.

The data quality watchdog is a built-in agent that notices when data stops flowing before your campaigns or reports suffer. Once a day it checks event volume per source, failed Connect runs and how many new profiles carry each identifier. It sends one plain-text email, but only when something is flagged.

Who uses it

Tenant administrators and data teams create the instance. You need the Built-in agents feature in your licence. The Connect check also needs the Connect package; without it the agent still runs, skips that check and says so in the email. The owner of the instance needs clearance to read the identifiers you list and a role that can count events.

Fields

ParameterWhat you enterDefault
recipientsThe people who get the email.Owner (you)
scheduleWhen the check runs, as a cron expression in the instance time zone.0 7 * * * (every day, 07:00)
sourcesRequired. The exact event source values to watch, for example edge for the web gateway, connector:<connector id> for a Connect batch connector or stream:<connector id> for a stream source. An event's Source in the console shows the value.None
drop_threshold_pctHow far a figure may fall below its comparison before it is flagged, in percent. Above 0 and up to 100; any other value falls back to 30 and the email says so.30
identifiersThe identifier keys whose fill rate to check.email, phone
entity_typeThe entity type whose new profiles are checked.customer
purposeThe processing purpose the event counts are read for. A tenant in strict privacy mode must set it.Empty
always_sendSend a short "all checks passed" email even when nothing is flagged.Off

Actions

  • Reads: event counts per source, recent Connect runs and counts of new profiles. It reads aggregates, not individual customers.
  • Sends: one plain-text email to the recipients you choose, with the subject "Data quality findings" and the date.

The three checks:

  • Event volume per source: yesterday's count against the same 24 hours on each of the previous four weeks. A source is flagged when yesterday had no events but an earlier week had, or when yesterday is clearly below the lowest earlier count. The email states yesterday's count and the four earlier counts, not a percentage.
  • Connect runs: every run in the last 24 hours that failed or ended partial, with its connector, start time, status, failed rows and error message.
  • Identifier fill rates: for each identifier, how many of the new profiles of the last 7 days have it, against how many of the new profiles of the 7 days before. The email states both pairs of counts.

The email lists "What I could not check": failed reads, sources or identifiers left out and "Connect runs not checked (no Connect licence)" when that check was skipped.

A check that cannot run is a finding, not silence: a failed read, a source with no events in any of the five windows, an identifier the entity type does not declare and an identifier that no new profile has in either week all make the email go out, whatever Always send says. Without any finding no email is sent unless Always send is on. When the schema description is too large to read in full, the email says the declarations were not checked and no identifier is reported as undeclared. Every figure is copied from Optima's own results and checked against what the agent read.

What it does not check

It does not detect schema changes or duplicate profiles; Optima does not record those today.

Limits

One run checks at most four sources and three identifiers, the first ones listed; the email names the rest. For more, create one instance per group of sources.

Examples

To try it, create the instance in dry run with sources set to edge, a cheap model and schedule a few minutes ahead. Open the run log to see what it would send, then switch the instance to live.

Common problems

  • No email today. Nothing was flagged and Always send is off, or the instance is stopped.
  • A source is flagged as having no events in any window. The value in sources is probably not the exact source of your events; check an event's Source in the console.
  • The email says a source could not be checked because of the privacy mode. Set purpose to a purpose the owner may read.
  • The email says the identifier declarations could not be checked. The list of entity types was too large to read in full; every identifier was still queried.
  • An identifier is flagged as not declared. The entity type does not declare it; check identifiers and entity_type. If it is flagged as missing from every new profile, the owner may have no clearance to read it, or nothing fills it.
  • The card is missing. Your licence has no Built-in agents feature.

On this page