by AgentPulse (LX AI)
Monitor your AI agents in production
AgentPulse reviews a pasted run for loops, drift, and odd tool calls. It does not map statutes and it does not watch production live. The notes below are general operations reading, not a product promise.
List the system prompt, the tools the agent can call, and the data each tool touches. You cannot guard what you have not enumerated. (OWASP LLM06 - over-reliance; LLM01 - prompt injection surface.)
Treat any external content the agent reads - emails, web pages, tool output - as untrusted. Strip instructions, require structured tool outputs, and keep a deterministic allow-list for sensitive actions. (OWASP LLM01.)
Secrets belong in env vars, never in prompts or logs. Scan env files and CI config for leaked keys before deploy, and rotate on hit. (OWASP LLM02 - sensitive information disclosure.)
Deletes, sends, and payments should be gated. The EU AI Act puts human oversight (Art. 14) at the center of high-risk obligations. (EU AI Act Art. 14.)
Every agent run needs a stable runId and a ruleset version so a bad output is traceable to a specific input and policy. This is your audit trail. (EU AI Act Art. 11 - technical documentation.)
Cap token spend per run and per day. Unbounded agents turn a bug into a bill. Surface HTTP 429 when the guard trips instead of silently degrading. (OWASP LLM10 - unbounded consumption.)
Policy logic should be reviewable in a pull request, not buried in a prompt. Version each ruleset so regressions are diffable. (EU AI Act Art. 9 - risk management.)
List the system prompt, the tools the agent can call, and the data each tool touches. You cannot guard what you have not enumerated. (OWASP LLM06 - over-reliance; LLM01 - prompt injection surface.)
Treat any external content the agent reads - emails, web pages, tool output - as untrusted. Strip instructions, require structured tool outputs, and keep a deterministic allow-list for sensitive actions. (OWASP LLM01.)
Secrets belong in env vars, never in prompts or logs. Scan env files and CI config for leaked keys before deploy, and rotate on hit. (OWASP LLM02 - sensitive information disclosure.)
Deletes, sends, and payments should be gated. The EU AI Act puts human oversight (Art. 14) at the center of high-risk obligations. (EU AI Act Art. 14.)
Want a fast, self-serve read on your own agent? Try AgentPulse
Every agent run needs a stable runId and a ruleset version so a bad output is traceable to a specific input and policy. This is your audit trail. (EU AI Act Art. 11 - technical documentation.)
Cap token spend per run and per day. Unbounded agents turn a bug into a bill. Surface HTTP 429 when the guard trips instead of silently degrading. (OWASP LLM10 - unbounded consumption.)
Policy logic should be reviewable in a pull request, not buried in a prompt. Version each ruleset so regressions are diffable. (EU AI Act Art. 9 - risk management.)
List the system prompt, the tools the agent can call, and the data each tool touches. You cannot guard what you have not enumerated. (OWASP LLM06 - over-reliance; LLM01 - prompt injection surface.)
Start with inventory: list every agent, its model providers, tools, memory stores, and human owners. Without inventory, checklists become theater. Next, classify data each agent may touch (public, internal, secret). Secrets should never be pasted into third-party playgrounds. Then enforce tool allowlists: prefer explicit registries over open-ended “browse the web and act” defaults.
Prompt-injection defenses belong at multiple layers. Treat untrusted retrieved text as data, not instructions. Separate system policy from user content. Log tool arguments. Add canary tests that try to override policy (“ignore previous instructions…”) and expect denial. Map each control to OWASP LLM Top 10 identifiers so audits stay comparable across quarters.
Runtime controls: rate limits, budget caps, sandboxing for code tools, and kill switches. Fair-use quotas that short-circuit before model calls protect both cost and safety. When AI keys are missing, return a clear not-configured error instead of silent mock success—operators must see the truth.
Post-run: retain traces long enough for incident review, redact secrets in logs, and define who triages severity. Red-team on a schedule; keep fixtures for known jailbreaks. Publish an internal “What this checklist is NOT” statement: it is not a legal opinion, not SOC 2 evidence by itself, and not a promise of zero incidents.
Use this page as a living document. Update RULESET dates when frameworks change. Prefer primary references over blog summaries when you cite requirements.
Not legal advice, not a penetration test report, and not proof of zero residual risk.
Document the control owner, the last test date, the fixture path, and the expected fail-closed behavior when credentials or quotas are missing. Record whether results are rule-based or model-assisted, and never present demo output as live AI. Prefer primary standards links over secondary summaries. Keep customer metrics empty until verified. Re-run this checklist after framework updates and after major prompt or tool-registry changes. Store evidence in tickets with timestamps. Reject efficacy language such as absolute guarantees.
Document the control owner, the last test date, the fixture path, and the expected fail-closed behavior when credentials or quotas are missing. Record whether results are rule-based or model-assisted, and never present demo output as live AI. Prefer primary standards links over secondary summaries. Keep customer metrics empty until verified. Re-run this checklist after framework updates and after major prompt or tool-registry changes. Store evidence in tickets with timestamps. Reject efficacy language such as absolute guarantees.
Document the control owner, the last test date, the fixture path, and the expected fail-closed behavior when credentials or quotas are missing. Record whether results are rule-based or model-assisted, and never present demo output as live AI. Prefer primary standards links over secondary summaries. Keep customer metrics empty until verified. Re-run this checklist after framework updates and after major prompt or tool-registry changes. Store evidence in tickets with timestamps. Reject efficacy language such as absolute guarantees.
Document the control owner, the last test date, the fixture path, and the expected fail-closed behavior when credentials or quotas are missing. Record whether results are rule-based or model-assisted, and never present demo output as live AI. Prefer primary standards links over secondary summaries. Keep customer metrics empty until verified. Re-run this checklist after framework updates and after major prompt or tool-registry changes. Store evidence in tickets with timestamps. Reject efficacy language such as absolute guarantees.
Document the control owner, the last test date, the fixture path, and the expected fail-closed behavior when credentials or quotas are missing. Record whether results are rule-based or model-assisted, and never present demo output as live AI. Prefer primary standards links over secondary summaries. Keep customer metrics empty until verified. Re-run this checklist after framework updates and after major prompt or tool-registry changes. Store evidence in tickets with timestamps. Reject efficacy language such as absolute guarantees.
Document the control owner, the last test date, the fixture path, and the expected fail-closed behavior when credentials or quotas are missing. Record whether results are rule-based or model-assisted, and never present demo output as live AI. Prefer primary standards links over secondary summaries. Keep customer metrics empty until verified. Re-run this checklist after framework updates and after major prompt or tool-registry changes. Store evidence in tickets with timestamps. Reject efficacy language such as absolute guarantees.