The Agent Development Lifecycle 101 by Harrison Chase

Agent Development Lifecycle (Harrison Chase, LangChain)

The talk presents an end-to-end process for building and operating AI agents reliably at scale. The hardest part is not getting a demo working locally—it’s achieving reliable behavior, good performance, and safe operation for many users.

---

Core Lifecycle Stages

1) Build

Choose how you build agents using varying levels of abstraction:

• Frameworks Standardize model inputs/outputs, tools, prompts, retrieval (examples: LangChain, LlamaIndex, CrewAI, Google ADK, OpenAI Agents SDK, AWS Swarm).

  • Tradeoff: frameworks can obfuscate what happens under the hood (e.g., multiple LLM calls hidden inside a chain).

• Runtimes Focus on operational concerns like state management, durable execution, and human intervention (examples: LangGraph runtime, Temporal).

• Harnesses Higher-level orchestration for complex agent behavior, including state management, file-system context, summarization/compaction, human-in-the-loop UX, sub-agents, skills (examples: Deep Agents, Claude Agents SDK).

  • “Coding agent harnesses” (e.g., CodeX/Open Code/Pi) are described as close cousins to developer harnesses.

No-code / Agents-as-files direction

Agent definitions increasingly can be represented as files rather than code:

• Prompts as text/markdown (e.g., agents.md)
• Tool connections via config/JSON (e.g., MCP config)
• Skills as mostly markdown with optional scripts
• Example referenced: Vercel’s eve.dev as an “agents-as-files” approach

---

2) Test (Evaluation)

Define testing as: run the agent on inputs, score behavior, and compare versions.

Two evaluation styles discussed:

• Regression testing: don’t get worse; aim for consistent pass rates
• Benchmark hill-climbing / progress evaluation: improve performance toward a target

Key components:

• Datasets

  • expected tasks
  • real edge cases from traces/dogfooding
  • synthetic generation

• Judging criteria

  • when possible: ground truth (e.g., classification)
  • when not: criteria-based judging (e.g., coding agents evaluated via unit tests rather than only file diffs)

• Metrics/experiments

  • correctness vs ground truth
  • criteria checks and policy compliance
  • combine metrics into experiments to catch regressions and decide readiness

Key points:

Datasets + metrics are application-specific and require product judgment.

Example for complex, long-running agents:

• Terminal Bench 2 (runs tasks in sandboxes using the Harbor framework) Emphasizes agent testing differs from LLM app testing because agents can run 15–30 minutes and produce artifacts / interact with environments.

---

3) Deploy

Deployment focuses on agent runtime/operational requirements:

• durable execution (resume if the agent fails mid-run)
• persistence & memory (thread-level plus longer-term)
• streaming (tokens and/or tool calls/thought process)
• human-in-the-loop at scale (pause/resume in production)
• sandboxes for safe execution—especially for code-generating agents

Virtual file systems

Instead of spinning up full file systems, store context in scalable backends (DB/S3/etc.) while exposing filesystem-like operations (read/write/ls/grep/glob).

Examples mentioned:

• Using Box and Notion as virtual file systems
• Building a prompt & context hub to store version prompts, skills, and instructions

---

4) Monitor

Monitoring is primarily observability + understanding what the agent did:

• Tracing
  • steps, inputs, outputs, tool calls, sub-agent calls
  • crucial for debugging: many failures come from missing/incorrect context
• Feedback signals
  • thumbs up/down and user-provided feedback
• Online evals
  • run evaluators over production traces (often without ground truth)
  • example concept: “perceived error” inferred from user reactions (e.g., “you messed up”, user-corrected code)
  • described as training a small model to detect this signal
• Dashboards
  • aggregate latency, time-to-first-token, feedback, and online-eval scores

---

5) Govern

Governance wraps the whole lifecycle:

• cost controls and cost visibility (including internal usage)
• tool access controls (who the tool is used on behalf of)
• audit trails (which tools were called and by whom)
• human approval requirements for sensitive actions
• tool/agent discoverability
• managing context/skills sharing

At scale:

• standardization via shared build frameworks and shared evaluation frameworks
• a centralized platform/team typically handles deployment infrastructure and shared observability

---

How LangChain Fits (Product/Platform Mapping)

The talk maps tools to lifecycle stages:

• Build tooling

  • Deep Agents (agent harness; recommended for starting)
  • LangChain (lean/minimal agent framework; used to build harnesses)

• Runtime

  • LangGraph (runtime under harnesses)

• No-code build

  • LangSmith Fleet (UI-based agent builder; natural language driven)

• Test / Deploy / Monitor / Govern

  • LangSmith platform:
    • datasets, evals, experiments
    • Harbor integration for sandboxed evals
    • deployments: agent server with production-ready endpoints (streaming, A2A/MCP, human-in-loop, short/long-term memory, etc.)
    • sandboxes (framework-agnostic code sandboxes)
    • context hub (store agent files like prompts/skills; used via virtual filesystem approach)
    • monitoring (tracing, online evals, dashboards, user feedback)
    • governance: LLM Gateway (private preview) for cost controls + visibility + privacy

“Traces are central”

LangSmith emphasizes that observability via traces is needed to:

1. observe behavior
2. measure improvements
3. iterate reliably

---

Built Components Highlighted

• Smith DB: purpose-built database for agent traces to make large-scale trace storage fast.
• LangSmith Engine (self-improving agent automation)
  • runs in the background over traces (every ~6 hours)
  • clusters traces into issues (issue board)
  • suggests/fixes prompts or code changes
  • adds examples to datasets, creates online evals, and monitors improvements
  • explicitly aims to reduce “whack-a-mole” operational burden

---

Key Q&A / Tutorials-Style Takeaways (Notable Mentions)

• Vercel eve.dev / agents as files: aligned with the “agents.md + file-system definition” direction.
• Terminal Bench 2 + Harbor: Harbor positioned as useful for building custom benchmarks.
• Token-count control inside frameworks
  • store large tool outputs to files; only pass recent segments back to the model
  • summarization thresholds
  • compact/remove long generated inputs while preserving reload ability from storage
  • caution about maintaining compatibility with prompt caching
• “Is there a standard pass rate?”
  • described as application-specific; no universal industry metric
• Testing novel experiences
  • use another LLM/agent to generate semantically similar scenarios and compare outputs (example: LangFuzz)
  • focus on eval environments (synthetic, resettable state for stateful systems like Linear/Jira)
• Monitoring drift / prompt changes
  • Engine can suggest fixes; “insights” described as hierarchical clustering to detect usage/topic drift over time
• Online eval speed vs agent runtime
  • online eval runs after the agent; doesn’t slow the agent run, though it may increase overall compute/cost
• Human input necessity
  • human review still expected when success criteria aren’t clear; annotation queues help SME-driven evaluation at scale
• Agent beyond chatbot
  • requires better UI/UX and likely “ambient/event-triggered” background agents

---

Main Speakers / Sources

• Primary speaker: Harrison Chase (co-founder & CEO of LangChain)

• Referenced external sources/frameworks/companies (as examples):

  • Vercel (eve.dev), Terminal Bench 2, Harbor, LangChain/LangGraph, LangSmith, Deep Agents, Temporal, OpenAI Agents SDK, AWS Swarm, LlamaIndex, CrewAI, Google ADK, Claude Agents SDK
  • CodeX/Open Code/Pi (coding agent harnesses)
  • Box/Notion as virtual file system backends