Campaign evaluator
A built-in agent that emails you when a campaign has been sent and a full evaluation of its results after a delay.
The campaign evaluator is a built-in agent that reports on your campaigns without you having to open them. It sends two emails per campaign: a short note about an hour after the campaign has been sent, and a full evaluation once enough time has passed for the results to come in.
Who uses it
Campaign managers and tenant administrators create the instance. The people who receive the emails can be anyone on your team or any email address you enter. You need the Built-in agents feature in your licence and the Campaign package with its campaigns, journeys and attribution features. An account that excludes journeys or attribution does not see this agent.
Fields
| Parameter | What you enter | Default |
|---|---|---|
recipients | The people who get the emails. | Owner (you) |
evaluation_delay | How long after the campaign finished the full evaluation is written. | 168h (seven days) |
campaigns | The campaigns to evaluate. Leave empty to evaluate every campaign. | Any |
conversion_event | The event type that counts as a conversion. Leave empty to skip the attribution part. | Empty |
Actions
- Reads: the campaign, its runs, its results per channel, the control-group lift, the per-offer responses, attribution for your conversion event, journey results and a few earlier campaigns of the same channel and type for comparison. It reads aggregates, not individual customers.
- Sends: the emails below to the recipients you choose.
About an hour after a one-off campaign has been sent (so that the delivery results are in) you get a plain-text note of at most 12 lines: profiles matched, sends published, duplicate persons skipped, the count of each step per channel (queued, sent, failed, bounced) as of then, and the error if the run failed. It ends by telling you when the full evaluation follows. It does not judge performance, because it is too early. If the campaign sent nothing, you get instead a note of four lines at most (that nothing was sent, the profiles matched and the error), and no evaluation follows.
After the evaluation delay you get the evaluation, at most 30 lines:
- Result: a one-sentence verdict against the earlier campaigns, naming the measure and giving both values.
- Funnel: the count of each step as returned, with the rate where one is available.
- Against the control group: the lift, or "no control group".
- What worked and What did not: up to three points each.
- Split tests: the winners and their measure, when the campaign has split tests.
- Next time: up to two concrete suggestions.
- Not checked: what the agent could not verify.
Journey and recurring campaigns do not complete, so they are evaluated when they stop. A campaign is stopped automatically when one of its stop conditions is met. Pausing or archiving a campaign does not start an evaluation. A one-off campaign is evaluated when it completes; a one-off stopped before it completes is not evaluated. A stopped campaign is always evaluated, even when it sent nothing. For a journey campaign the email has a "Journey drop-off" section: per journey, the nodes where the most runs exited or failed, with their counts.
Every number in the emails is checked against what the agent read. If the agent cannot back a figure up, the run fails and nothing is sent.
Examples
To try it, create the instance with an evaluation_delay of 2h, run a small one-off campaign, and look at the run log: the short note appears an hour after the campaign has been sent and the evaluation an hour after that. Create the instance in dry run first so that the run log shows the emails instead of sending them, then switch it to live and set the delay back to seven days.
Common problems
- No email after a campaign. The campaign is not in the
campaignslist you chose, it is a journey or recurring campaign that has not stopped, or the instance is stopped. - The evaluation has no attribution part. The
conversion_eventis empty. - The run failed with "numbers that no tool returned". Try a stronger model or look at the run in the run log.
- The evaluation lists parts as not checked. The agent's owner may lack access to some of the data. Create the instance as someone who may read it.
- The card is missing. Your licence has no Built-in agents feature, or no campaigns, journeys or attribution feature.