What should agent monitoring tell you?

Agent execution monitoring checks whether an expected job started, made reported progress and finished within its declared limits. It is useful when nobody is watching the run. It complements tracing and output evaluation; each answers a different question.

Three questions, three kinds of evidence.

A trace can help explain a runtime incident. An output check can reject a report even when the process exited normally. Start with the question you need answered, then choose the smallest signal that supports it. For a primary reference on trace structure, see OpenTelemetry’s traces documentation.

Write an execution contract.

1. Expected start: Is the job recurring? Record the same schedule and timezone used by its launcher, with a grace period for normal jitter.
2. Maximum runtime: Decide when a still-active run needs investigation, allowing for legitimate external latency.
3. Meaningful progress: Define a completed unit of work, such as a saved record or finished research phase. Emit progress after that unit succeeds.
4. Completion: Separate normal process exit from a named result such as report_saved. Use a configured verifier if independent evidence is needed.
5. Response: Choose who receives the incident and how they inspect it. Keep recovery manual until runtime cooperation has been tested.

For a daily report, “started at 07:00 UTC within five minutes, completed within fifteen minutes, and report persisted” is more actionable than “agent healthy.” Watchdog can monitor the declared expectations; your code must report the relevant events.

Silence is a symptom.

A missing completion could mean a dead process, a deadlock, a blocked dependency or lost telemetry. Watchdog does not inspect your operating system or Kubernetes pods. An unscheduled run whose start never arrives is invisible. A recurring run can be missed only if its expected schedule was registered.

The SDK queues telemetry without blocking your application on network delivery. The queue and retries are bounded, and a killed process may lose events. Inspect delivery counters during setup and compare incidents with your runtime logs.

Check SDK delivery.

The run context emits a start and normal completion, or reports failure while preserving an exception. The outer client drains its bounded queue. A successful SDK construction or printed message does not prove delivery: compare the received events in Runs with the registered job and workspace key. Use WATCHDOG_URL=https://withwatchdog.com and a telemetry key, never a login password or session cookie. The import is watchdog_agent, not the unrelated filesystem package watchdog.

# Inside the Watchdog client context, after the run finishes:
if not watchdog.enabled:
    raise RuntimeError("Set WATCHDOG_API_KEY first")
if not watchdog.flush(timeout=10):
    raise RuntimeError("Telemetry did not drain")
stats = watchdog.stats
if stats["dropped"] or stats["sent"] != stats["queued"]:
    raise RuntimeError("Inspect telemetry delivery")

The errors counter includes transient attempts recovered by retry; compare sent, queued and dropped after flushing. A rejected event or exhausted quota means the monitoring record may be incomplete. Progress messages and explicit metadata are transmitted: do not include prompts, secrets or customer payloads.

Read configuration, evidence and results separately.

A configured check can be awaiting evidence. Monitoring the instrumented scope means Watchdog received relevant signals; it never establishes that all tools or model requests were instrumented. Partial coverage identifies known gaps. Coverage unavailable or degraded identifies a paused monitor, known rejected telemetry or exhausted quota. A run can have completed normally while its monitoring record is incomplete.

Missing optional instrumentation does not fail a valid lifecycle-only run. Tool-loop findings concern reported calls with usable fingerprints: no finding does not establish that unobserved tools were healthy. An attached adapter, a framework name in job settings or one event is not evidence of complete application coverage.

Activity is not meaningful progress.

A model can call tools repeatedly without finishing useful work. Tool calls do not reset Watchdog’s progress timer; run.progress() does. There is no automatic agent heartbeat emitter in the base SDK. New jobs leave progress monitoring disabled. Start with lifecycle events and a runtime deadline. After adding meaningful milestones, explicitly enable progress monitoring and choose a timeout that fits the interval between them. The timer starts at run start; sending progress alone does not enable it.

Do not emit “still alive” as business progress merely to silence an alert. Conversely, report legitimate work as it finishes so a productive batch does not look stuck. The tool-loop guide explains argument fingerprints and repeated-work caveats.

In the job’s Additional checks, enable Monitor meaningful progress milestones and choose a timeout longer than the expected gap between milestones. Existing jobs retain saved policies; explicitly turn off a previously configured timeout for lifecycle-only monitoring. Policy changes apply to new runs.

Know what cost and outcome signals establish.

A named outcome is an attestation from your application, even when every required name was reported. A signed external verifier checks only its configured predicate; it cannot judge every property of an answer. A pending, failed or unavailable verifier is not a passing result. A lifecycle completion alone does not establish business success.

Cost checks sum the cost_usd values your code supplies. Missing cost is unknown, not $0; an explicit zero is a measurement. Costs can be partial or estimated, and a reported cost for one request does not establish complete run accounting. Token counts alone do not calculate spending. An out-of-band alert or cancellation request cannot guarantee an exact spending cutoff.

# Inside with watchdog.run(...) as run:
# Report each milestone or outcome only after its work succeeds.
run.progress("Research complete", completed=3)
run.outcome("report_saved", success=True)
# Only if you know the cost of this request:
run.usage(model="configured-model", cost_usd=0.02)

These are independent optional signals, not a claim that this snippet saved a report. Use run.tool_call() around an actual tool operation as shown in the tool instrumentation guide.

Watchdog also displays seven-day SLO calculations. Aggregate SLO target breaches do not currently generate separate alerts. Per-run incidents and aggregate reliability targets should be interpreted separately.

Recovery requires cooperation from your runtime.

Cancellation, retry and model fallback are optional advanced controls. The SDK can observe a cancellation only when cooperating application code stops; blocking calls cannot be forcibly interrupted. Retry requires your own controller to run the work again. A fallback model name is a request to that controller, not a model switch performed by Watchdog. A configured endpoint or accepted command is not evidence of a successful recovery. Leave these controls off for basic monitoring; existing saved policies retain their behavior.

Framework adapters and their scope.

The public SDK examples cover plain Python and the supported framework adapters. Keep framework dependencies optional; use the base SDK for lifecycle-only setup.

python -m pip install 'watchdog-agent-sdk[openai]==0.2.0'

SDK 0.2.0’s tested OpenAI extra pins openai-agents==0.22.0 and openai==3.8.0. It supplies WatchdogHooks to an existing Runner call and observes that call’s supported function-tool and model completion hooks. Hosted provider tools, calls outside that Runner, interrupted calls and errors converted into result strings may remain unobserved. Hooks do not infer meaningful progress, required outcomes or dollar cost. Do not wrap a tool twice when the adapter already records it. Read the adapter verification notes before upgrading dependencies.

A practical first policy.

Instrument one low-volume job. Start with lifecycle events, a realistic deadline and one owner-controlled alert destination. Use synthetic failures to check what the system actually records. Add progress, tool, cost and outcome signals only when you can define their meaning. The quickstart follows that path.