Field guide

Break a synthetic job. Inspect the evidence.

Reproduce successful execution, a silent run, a repeated tool cycle and legitimate repetition using the actual SDK and an isolated local Worker database.

Code and examples reviewed September 5, 2026

The Agent Monitoring Failure Lab exercises the actual Python SDK against the compiled Watchdog Worker and a fresh local D1 database. Four small synthetic cases separate a real detector finding from legitimate repeated work. No model provider, customer job or paid inference is involved.

Reproduce it from the supplied source.

Use Node 22.13 or later, pnpm and Python 3.10 or later. From the repository root:

pnpm install --frozen-lockfile
pnpm build
python3 scripts/failure-lab.py --output work/failure-lab-evidence.json

The harness creates a temporary database and random local credentials, runs the compiled Worker, and emits SDK events. It kills only its own synthetic child process. A separate process runs the supplied local scheduler through the authenticated tick API. There are no real alert destinations and no production API requests.

What is configured?

All four jobs are unscheduled. The silent case has an eight-second runtime limit; the others use 600 seconds. Progress timeouts are 600 seconds, the loop threshold is four repetitions, and tool limits allow the small fixtures. Cancellation and agent retry are off; runs are not cancellable. Outcome verification and all alert channels are disabled.

Success: start, one synthetic tool completion and normal completion.
Silent execution: a child process flushes its start event, then the harness kills that child. No terminal event is invented.
Repeated cycle: search → read repeats four times with the same arguments and no progress.
Legitimate repetition: eight calls process eight distinct record IDs. The harness checks for false positives while the run is still open, then completes it.

Expected and observed behavior.

Observed in an isolated local run on 2026-09-05. Environment: compiled local Worker, isolated D1, independent local scheduler; Python 3.12.14, Node v24.19.0, Wrangler 4.92.0.

Four fixtures, not a production accuracy benchmark
Case	Expected	Observed
Success	Normal completion; no incident	Success; no incident
Silent execution	Runtime incident after an independent check	runtime_exceeded; run remains running
Repeated cycle	Four identical search/read cycles produce a finding	tool_loop; two-tool sequence repeated four times
Legitimate repetition	Distinct arguments do not form the repeated pattern	Eight distinct fingerprints; no incident while open; then success

The final database contained four runs, two incidents and zero actions. Extra scheduled checks created no duplicate incidents. Test credentials and temporary state were removed. The local evidence file contains timestamps, policies, event summaries and source/build hashes.

Why the stopped-scheduler control matters.

Before enabling the local scheduler process, the harness waits past the silent run’s deadline. No overdue incident should be persisted just because wall time passed. Tool repetition can already be detected during ingestion. The independent process then runs scheduled checks while the harness makes no application API reads.

This distinguishes an ingestion-time finding from work that requires an independent check. The final snapshot must contain a new runtime incident and an advanced scheduler heartbeat.

What this lab does not establish.

It uses the supplied local scheduler process. It does not verify native Cloudflare Cron availability or production capacity.
Alert destinations are disabled. It does not test webhook transport, Slack, Resend, PagerDuty or a person receiving an alert.
It does not test a missing scheduled start, model spending, independent output verification or every detector.
The monitor sees missing completion, not the harness’s operating-system signal. A real network failure could look similar.
Different arguments are a useful negative control, not proof that all legitimate repetition avoids findings.

For alert delivery tests, use the source checkout’s docs/DEMONSTRATION.md and its separate synthetic HTTPS collector procedure. For a real job, follow the alert verification steps and keep recovery off.