What should agent monitoring tell you?

Agent execution monitoring checks whether an expected job started, made reported progress and finished within its declared limits. It is useful when nobody is watching the run. It complements tracing and output evaluation; each answers a different question.

Three questions, three kinds of evidence.

A trace can help explain a runtime incident. An output check can reject a report even when the process exited normally. Start with the question you need answered, then choose the smallest signal that supports it. For a primary reference on trace structure, see OpenTelemetry’s traces documentation.

Write an execution contract.

1. Expected start: Is the job recurring? Record the same schedule and timezone used by its launcher, with a grace period for normal jitter.
2. Maximum runtime: Decide when a still-active run needs investigation, allowing for legitimate external latency.
3. Meaningful progress: Define a completed unit of work, such as a saved record or finished research phase. Emit progress after that unit succeeds.
4. Completion: Separate normal process exit from a named result such as report_saved. Use a configured verifier if independent evidence is needed.
5. Response: Choose who receives the incident and how they inspect it. Keep recovery manual until runtime cooperation has been tested.

For a daily report, “started at 07:00 UTC within five minutes, completed within fifteen minutes, and report persisted” is more actionable than “agent healthy.” Watchdog can monitor the declared expectations; your code must report the relevant events.

Silence is a symptom.

A missing completion could mean a dead process, a deadlock, a blocked dependency or lost telemetry. Watchdog does not inspect your operating system or Kubernetes pods. An unscheduled run whose start never arrives is invisible. A recurring run can be missed only if its expected schedule was registered.

The SDK queues telemetry without blocking your application on network delivery. The queue and retries are bounded, and a killed process may lose events. Inspect delivery counters during setup and compare incidents with your runtime logs.

Activity is not meaningful progress.

A model can call tools repeatedly without finishing useful work. Tool calls do not reset Watchdog’s progress timer; run.progress() does. There is no automatic agent heartbeat emitter in the base SDK. Choose a progress timeout that fits the interval between genuine milestones, or make it long enough for a lifecycle-only job.

Do not emit “still alive” as business progress merely to silence an alert. Conversely, report legitimate work as it finishes so a productive batch does not look stuck. The tool-loop guide explains argument fingerprints and repeated-work caveats.

Know what cost and outcome signals establish.

A named outcome is an attestation from your application. A signed external verifier checks its configured predicate; it cannot judge every property of an answer. Transport failure leaves an outcome unverified. A lifecycle completion alone does not establish business success.

Cost checks sum the cost_usd values your code supplies. Costs can be partial or estimated; token counts alone do not calculate spending. An out-of-band alert or cancellation request cannot guarantee an exact spending cutoff.

Watchdog also displays seven-day SLO calculations. Aggregate SLO target breaches do not currently generate separate alerts. Per-run incidents and aggregate reliability targets should be interpreted separately.

A practical first policy.

Instrument one low-volume job. Start with lifecycle events, a realistic deadline and one owner-controlled alert destination. Use synthetic failures to check what the system actually records. Add progress, tool, cost and outcome signals only when you can define their meaning. The quickstart follows that path.