Why do AI Agents Fail

Summary

The video explains that AI agents usually fail not because the underlying model is insufficiently capable, but because of weaknesses in the surrounding system: its goals, data, tools, state management, security, and monitoring.

An agent harness coordinates a loop of planning, acting, calling tools, checking results, and repeating until the task is complete. Agents may use multiple models, selected according to a task’s complexity, context, latency, and cost.

Seven common failure modes and how to address them

1. Vague goals — The agent lacks clear boundaries, escalation rules, or a definition of what “done” and “good” mean. Define measurable objectives and create a “golden” evaluation set of at least 20 varied cases, including edge cases, to test changes. Use deterministic workflows for rule-based tasks, agents where judgment is needed, and human involvement where accountability requires it.

2. Bad context — Missing, inaccurate, outdated, irrelevant, or improperly permissioned data can produce confident but incorrect answers. Give agents their own identities and least-privilege access. Keep knowledge owned, current, and versioned; retrieve only relevant information; and evaluate retrieval quality and whether answers are grounded in valid sources.

3. Tool misfires — Agents may select the wrong tool, provide incorrect parameters, or fail to notice a tool error. Prefer a smaller set of clearly described, non-overlapping tools with typed, validated inputs and specific error responses. A managed gateway can standardize authentication, logging, and retries; circuit-breaker patterns can help prevent cascading failures. Evaluate tool selection and call accuracy.

4. Poor reasoning — Plans may be flawed, assumptions may go unchecked, or errors can compound across long, multi-step tasks. Allow agents to ask clarifying questions, express uncertainty, and escalate rather than guess. Check plans before execution, use evaluators or reviewers, and choose the model and reasoning depth appropriate to each step—for example, more reasoning for planning and a smaller model for routine work.

5. Lost state — Long-running agents can forget completed work, use outdated status, repeat actions, or stop partway through a task. Keep authoritative state in a system-managed store rather than relying on the request or model context. Use concise state summaries, checkpoints, action IDs, idempotent retries, and session isolation.

6. Hijacking and excessive privileges — Indirect prompt injection can arrive through websites, documents, email, or tools. Reduce risk with dedicated agent identities, least privilege, prompt-injection protections, and clear separation between data and instructions. Use proportionate human approvals, red-team testing, and broader security monitoring across identity, data, network, and tool activity.

7. Flying blind — Agent behavior and costs can drift as models, APIs, policies, and user needs change. Track each prompt, model call, tool call, token, cost, and outcome. Continuously evaluate live signals and alert on changes in quality, latency, and cost.

Continuous improvement

The speaker recommends turning production traces and feedback into new test cases, expanding the golden evaluation set, grouping recurring errors, and changing one system component at a time. Validate changes with regression and red-team tests in CI/CD, require human approval before promotion, and use versioned deployments with rollback.

Immediate safeguards, or “circuit breakers,” are still needed alongside long-term improvement. Improvement should not be applied unchecked directly in production.

Key takeaway

Define readiness and quality, then evaluate against those definitions throughout development and operation. Evaluation provides the feedback needed to detect failures and improve the system without relying on blind trust in the model.

Guides, reviews, or tutorials

No product review or step-by-step tutorial is presented. The video includes demonstrations of Microsoft Foundry features such as evaluators, model routing, guardrails, and agent optimization.

Main speaker/source

John Savill, presenter on John Savill’s Technical Training.