Detecting agent loops and drift in production

Why latency-only alerts fail for agents

Classic service monitoring grew up around a request that starts, ends, and returns a status code. An autonomous agent run does not obey that contract. It can call tools in a tree, retry the same tool with slightly different arguments, abandon a plan mid-way, or keep going long after a human would have stopped. A green p99 latency chart can sit next to a runaway planner that burns tokens while making no progress.

OpenTelemetry's GenAI conventions increasingly treat agent work as first-class telemetry: invoke_agent style spans, tool execution children, and optional content capture when privacy policy allows. That plumbing helps, but conventions alone do not tell you which signals deserve a page. You still need product rules for loops and drift that match how your agents actually behave.

AgentPulse sits in that product layer. Paste a sample run or behaviour log, choose a watch focus, and get a report that highlights likely loops, drift candidates, and anomalies. It is not a full OTLP backend and it is not a formal security audit. Detection quality depends on the events you provide.

A four-step watch pattern

- Ingest events. Capture decisions, tool calls, results, retries, and abandonment reasons per run. Prefer structured fields over free-text-only logs so you can count repeats without NLP heroics. - Define normal for that agent. Typical plan length, tool mix, retry count, and duration band — per agent identity, not a global fleet average that mixes chatbots with planners. - Detect anomalies. Repeated near-identical steps; run length far above the band; a never-before-seen tool in production; escalating retries followed by abandonment; output length or refusal rate moving away from baseline. - Report with evidence. A watch entry must link to a replayable trail. If an on-call engineer cannot open the run and see the decisions in order, the alert will be ignored within a week.

Signals worth paging on

- Repeat count for the same tool with the same or near-identical arguments. - Step count or duration above that agent's usual band after excluding known long tasks. - A tool that has never been called for this agent appearing in production traffic. - Escalating retries followed by abandonment without a user-visible answer. - Sudden shifts in tool mix after a model or prompt rollout — often drift or a bad release, not "the users got harder".

Do not page on every slow run. Long runs that still advance are not loops. Alert fatigue is how monitoring products die. Prefer rate-of-change against an agent-specific baseline, then require a trail so each alert can be closed with evidence.

Baselines break — record why

A model change, a prompt change, a new tool, or a change in traffic mix will move your numbers. If you do not record those events, every release looks like mysterious drift. Keep a simple change log next to the metrics: which model, which prompt hash, which tools were enabled, and when traffic composition shifted. Consumers of producer-emitted scores (trust, drift, scan) in emerging OTel proposals are already warned that method tokens matter — when the scoring method changes, re-anchor correlation rather than treating the jump as agent misbehaviour.

What monitoring is not

Monitoring tells you something happened. A runtime step cap or guardrail stops it. Confusing the two leads to over-claiming. AgentPulse does not guarantee that alerts will always fire in time to prevent user impact. We do not claim 100% detection. We do not claim you will never miss a loop. Those honesty lines exist because FTC-style marketing rules punish fabricated certainty, and because operators deserve tools that state their limits.

Privacy and retention

Traces often contain user content. Keep them long enough to investigate, short enough to honour what you told users in the privacy policy. Default short retention, export on demand, and access controls consistent with GDPR Art. 32 framing are the floor — not a certification badge you can stamp on a sales deck.

How AgentPulse fits in a toolchain

Use your existing APM for the services around the agent. Use OpenTelemetry (or equivalent) for span plumbing if you already invest there. Use AgentPulse when you need a focused watch report on loops, drift, and anomalous tool use with a replayable narrative — especially early, when standing up a full agent control plane is overkill. Enterprise can add BYOK and DPA paperwork; Free and Demo stay labeled so nothing pretends to be live AI when the key is missing.

Worked example (synthetic, not a customer story)

Imagine a planner agent that normally takes 6–10 tool steps. Overnight, runs stretch to 40+ steps and the same search tool is called with near-identical queries. Latency is still "fine" because each tool call returns quickly. A loop detector that watches repeat counts and plan length would raise a watch entry; a latency-only dashboard would stay quiet. Opening the trail would show the repeating search block. That is the difference between agent observability and classic APM.

Now change the prompt to allow a new research tool. Tool mix shifts. Without a change log, that looks like drift. With a change log, it is an expected baseline reset. Good monitoring products make that distinction boring and explicit.

Operational checklist

- Emit per-run decision events (tool, args hash, result class, retry count). - Store a replayable ordered trail for every paged incident. - Baseline per agent identity; reset on model/prompt/tool changes. - Separate detection (watch) from enforcement (guardrails / step caps). - Never market 100% detection or never-miss assurance. - Treat traces as potentially sensitive personal or business data.

Instrumentation tips that reduce false positives

Hash tool arguments before comparing them. Exact string equality on free-form JSON creates noise when timestamps or request IDs differ. A stable fingerprint of the semantic fields (query text, resource id, action name) keeps loop detection honest without storing full payloads forever.

Separate user-initiated retries from agent-initiated retries. A human clicking "try again" is not the same failure mode as an agent re-entering the same planning node. Tag the initiator on each retry event so your watch rules can treat them differently.

Exclude known long workflows from the default duration page. Batch research agents may legitimately run for minutes. Put those agent IDs on an allowlist with a higher band, rather than silencing the whole fleet.

Version your watch rules the same way you version prompts. When a rule changes, annotate the dashboard so a spike is not misread as agent drift. Emerging producer-emitted score conventions already recommend method tokens for this reason — apply the same discipline to your own detectors.

On-call playbook sketch

- Open the watch entry and confirm the trail shows a real loop or drift pattern. - Check the change log for model, prompt, or tool enablement in the last 24 hours. - If confirmed loop, enable or tighten the runtime step cap for that agent identity. - If confirmed drift after a release, roll back or re-baseline with an explicit note. - If the trail is incomplete, fix instrumentation before raising more pages — alerts without evidence waste trust.

Document the close reason on every page. Patterns like "false positive due to traffic mix" teach the next shift what to ignore. Patterns like "prompt regression" teach engineering what to test before ship.

What AgentPulse will not invent

We will not invent customer logos, uptime percentages, or "reduced incidents by X%" claims. Social proof sections stay marked PENDING until real quotes exist. ROI numbers in content modules are templates. That is deliberate FTC-aligned restraint, not incomplete marketing.

Frequently asked questions

What is the difference between a loop and a long run?

A long run makes progress. A loop repeats a step, or a near-identical variant, without getting closer to the goal.

How do we avoid alert fatigue?

Alert on rate of change against that agent's own baseline, and require a replayable trail so each alert can be closed quickly.

What breaks a baseline?

A model change, a prompt change, a new tool, or a change in the traffic mix. Record those so you do not read them as unexplained drift.

Should a loop stop the agent?

That is a runtime guardrail decision, not a monitoring one. Monitoring tells you it happened; a step cap stops it.

How long should traces be kept?

Long enough to investigate, short enough to respect whatever you promised users about retention. Traces usually contain user content.

Is drift always bad?

No. Traffic changes can explain it. Drift is a prompt to look, not a verdict.

Does AgentPulse replace OpenTelemetry?

No. OTel is plumbing. AgentPulse is a focused watch-report product that can sit beside OTel-backed pipelines.

References

- OpenTelemetry — GenAI observability (2026) - OTel GenAI agent spans - NIST — AI Risk Management Framework

Try the product: AgentPulse

2026-09-26 · primary sources only · no fabricated traffic metrics · decision-support only · RULESET_VERSION 2026-09-26

Related articles

- Agent observability vs classic APM - Explore the full AI micro-SaaS topic matrix →