Skip to content
Optima docs
ConsoleIntegrationsData connectors

Dead letters and retries

How failed rows and messages are retried, counted and set aside, and where to see them.

Not every row or message is processed on the first try. This page explains what Optima does when something fails, and where you see it. You watch all of this on the Runs tab of Data connectors and on the Runs and Stream log tabs of a connector. Reading needs the integration_viewer role, or another role your administrator has given access to this area — see Licences and roles.

Batch connectors

A run goes through the whole file or query. A row that cannot be read, mapped or ingested is counted as failed but does not stop the run. The run ends with one of these statuses:

StatusMeaning
RunningIn progress.
SuccessFinished with no failed rows.
PartialAt least one row failed, including a run that stopped with an error after some rows had failed. The rest were imported. A file in which every row fails is also Partial.
FailedThe run stopped with an error before any row was counted as failed, for example because the system was unreachable.

Each run records Rows processed, Rows failed, Skipped, the Files involved and an Error. A source file is imported only once, so running again skips files already imported; fix the file and upload or name it again to retry the failed rows. A One-time job retries on later runs until one is fully successful.

Stream connectors

A message that fails to be ingested is retried. A message that keeps failing is set aside, so it no longer blocks the messages behind it, and the connector goes on with the next one. A single message that succeeds after a failure does not end up set aside. Set-aside messages are kept for 72 hours and then discarded. The console does not list or replay them.

A message that your transform script handles is treated differently, and is not retried. If the script returns None, the message is skipped and counted as skipped. If the script returns a string or fails with an error, the message is dropped, counted under Rows failed, and the reason is logged in the Stream log. The Stream log shows samples and errors, with the payload before and after the script, so you can see why. A run row of a stream connector is a listening session with live counts.

Examples

Rows fail in a nightly import. The run is Partial. Open the run in Runs, read the Error, correct the data or the mapping, and import the corrected file.

A stream message keeps failing. Check the Stream log on the Errors filter. If the cause is in your data, filter the message out with an Import filter or handle it in the script.

Common problems

  • Failures vanished after a few days. Stream log entries and set-aside messages are not kept indefinitely.
  • Rerunning does not retry. Files already imported are skipped.

On this page