Video summary

From Chaos to Choreography: Multi-Agent Orchestration Patterns That Actually Work — Sandipan Bhaumik

Main summary

Key takeaways

Educational

Main ideas, concepts, and lessons

  • Multi-agent “scaling” is not feature scaling

    • Going from 1 agent to 5 agents does not just add 4 more capabilities; it turns the problem into distributed systems coordination.
    • Complexity can grow ~25x due to coordination relationships and more failure/state synchronization paths.

  • Key lesson: treat multi-agent systems as distributed systems

    • Many failures are not due to the LLM or prompts, but due to architecture mistakes (race conditions, stale reads, missing coordination, weak observability).

  • War story: race condition caused by stale cache

    • A credit decisioning pipeline initially worked with one agent.
    • After adding four more agents, incorrect decisions occurred (e.g., risk ratings differed between agents).
    • Root cause:
      • The system used a cache layer between agents and the database.
      • Writes to PostgreSQL succeeded, but cache invalidation failed, so downstream agents read stale credit scores.
    • Lesson: Problems occurred at the architectural layer, not within the model.

Methodologies / patterns presented

1) Coordination choice: Choreography vs Orchestration

Choreography (event-driven, decentralized)

  • Agents are autonomous and communicate via events.
  • Pattern:
    • Agent A publishes an event (e.g., research_completed).
    • Agent B subscribes to relevant events, processes, and publishes next events (e.g., analysis_ready).
    • Agent C consumes those events, generates outputs, etc.
  • Properties:
    • Loosely coupled, high autonomy
    • Easier to add new agents (agents subscribe to new event types)
    • Scales well
  • Tradeoff / failure mode:
    • Debugging becomes difficult:
      • Which agent failed?
      • Did the event get consumed?
      • Was the event consumed twice?
  • Requirements to make it workable:
    • Bulletproof observability
    • Strong guarantees around event delivery and tracing

Orchestration (centralized workflow manager)

  • A workflow orchestrator controls the execution graph.
  • Pattern:
    • Orchestrator calls Agent A → waits → gets results
    • Orchestrator calls Agent B and C in parallel if needed → collects results
    • Orchestrator calls Agent D using combined outputs
  • Properties:
    • Orchestrator is single source of truth
    • Orchestrator manages:
      • State
      • Retries
      • Logging/observability
      • Execution order and dependency graph
    • Agents are “dumb”:
      • They only take input, do work, return output
  • Tradeoff:
    • Less autonomy for agents than choreography
  • When it’s preferred:
    • Complex dependencies
    • Need rollback/compensation
    • Need a single dashboard and deterministic debugging
    • “Workflow relatively stable” environments
  • Databricks implementation hint (as described):
    • Use LangGraph wired into an agent framework as the orchestrator
    • Or any workflow engine that supports DAGs and retry mechanisms

Decision framework (2 axes)

  • Axis 1: Workflow complexity (simple → complex)
  • Axis 2: Autonomy requirements (low → high)

Mapping:

  • Simple workflow + high autonomy → Choreography
  • Complex workflow + low autonomy tolerance → Orchestration
  • Top-right (complex + high autonomy) → Hybrid
    • Choreography + saga/compensation patterns (compensation/rollback for partial completion)
    • Mention of “packaged tooling” like Agent Bricks to reduce rebuilding orchestration patterns

2) State management: avoid race conditions with immutable state snapshots + versioning

What most people do first (and what breaks)

  • Shared mutable state
  • Multiple agents concurrently updating the same records
  • Risks:
    • Lost updates (“last write wins”)
    • Stale reads
    • Race conditions if transactions/locking are not handled explicitly

What works: immutable state evolution

  • Use immutable state snapshots with version numbers.
  • Pattern:
    • Agent A produces state version 1 (sealed/immutable; append-only logging).
    • Agent A hands version 1 to Agent B.
    • Agent B validates input contract/schema → produces version 2 (append-only insert; no updates to version 1).
    • Agent B hands version 2 to Agent C, etc.
  • Failures:
    • If an agent fails, you can roll back to the last good version.
    • Debugging becomes replayable: inspect state version lineage.

Data structure / code behavior described

  • State objects are frozen/immutable (Python concept).
  • Each state includes:
    • version
    • payload/data
    • created_by (who produced it)
  • A handoff function:
    • Validates schema / contract
    • Increments version and constructs the next immutable state
    • Executes the next agent with the immutable state

Why it prevents bugs

  • No concurrent modification to the same record
  • No stale reads from partially updated shared state
  • Clear lineage for “binary search” debugging through versions

3) Data contracts: enforce input/output schemas at agent boundaries

  • Problem:
    • Without contracts, agents may pass arbitrary or low-quality data downstream.
  • Pattern:
    • Each agent publishes what it outputs and what the next agent requires.
    • Example described:
      • Research agent outputs: findings, confidence score, sources, timestamp, etc.
      • Analysis agent requires: research output with defined types/fields.
      • Contract rule:
        • If confidence < 0.7, analysis agent rejects the handoff (fails early).
  • Databricks-based governance suggestion:
    • Register versioned schemas in Unity Catalog
    • Contracts are governed and versioned centrally

4) Failure recovery: design for failure with circuit breakers + saga/compensation

A) Circuit breaker pattern (fail fast; prevent cascading failures)

  • Pattern:
    • When Agent A calls Agent B:
      • Wrap calls with a circuit breaker.
    • If Agent B fails repeatedly (e.g., 5 times in a row):
      • Circuit opens → fail fast instead of timing out repeatedly.
    • After a cooldown (e.g., 60 seconds):
      • Circuit moves to half-open
      • Test with one request
      • If success → close circuit
      • If fail → re-open circuit and reset timer
  • Expected system behavior:
    • Graceful degradation:
      • Skip that agent
      • Use cached results
      • Or alert humans
    • Prevents one agent failure from cascading into whole-workflow failure
  • Databricks enforcement hint:
    • Enforce policies at serving layer using Model Serving / AI Gateway
    • Log open/close transitions for visibility (e.g., via MLflow)

B) Compensation pattern (Saga pattern) (rollback partial work)

  • Motivation:
    • Agents can fail mid-workflow; you need to undo partial effects.
  • Pattern:
    • Each agent implements two methods:
      • execute (do work)
      • compensate (undo/rollback)
    • Orchestrator tracks which agents completed successfully.
    • If later execution fails:
      • Orchestrator walks backward through completed agents
      • Calls compensate in reverse order
  • Example described (conceptual):
    • Analysis agent compensation deletes draft recommendations it wrote
    • Research agent compensation clears cached research data
    • Result:
      • System returns to the initial state
      • Avoids stuck workflows / partial transactions

Production architecture (how pieces fit together)

High-level “working production” workflow design

  • Orchestrator (single source of truth):
    • Holds the workflow engine and state versions
    • Calls agents according to DAG/graph dependencies
    • Manages observability and rollback logic
  • Agents:
    • Implement as functions/models (described as Unity Catalog functions or models)
    • Never call each other directly
    • Take inputs and return outputs
  • State store:
    • Immutable, append-only state snapshots (e.g., Delta tables)
    • State evolution tied to MLflow traces for replay/debugging
  • Serving layer protections:
    • Circuit breaker policies, retries/timeouts/rate limits enforced at serving/AI gateway layer
  • Tracing/evaluation:
    • MLflow traces capture inputs/outputs/latency/token usage
  • Compensation:
    • If an agent fails, orchestrator triggers compensation methods for already-completed steps

Databricks-specific implementation (as described)

  • Orchestration layer:
    • LangGraph + Mosaic AI agent framework
    • Agents implemented as Unity Catalog functions or registered models
  • Serving layer:
    • Databricks Model Serving / Function Serving
    • Circuit breaker behavior enforced via AI Gateway configuration
  • Data layer:
    • Delta Lake stores:
      • immutable versioned state snapshots (append-only rows)
      • workflow/customer data needed for execution
  • Governance/metadata:
    • Unity Catalog for access control, lineage, audit trail
  • Observability:
    • MLflow for per-agent tracing and LLM evaluation/metrics
  • Packaging suggestion:
    • Agent Bricks to package orchestration patterns for common use cases

Final takeaways emphasized

  • Agent chaos is inevitable once you move beyond one agent: coordination problems, race conditions, cascading failures.
  • You must choose between choreography vs orchestration (or hybrid) based on dependency complexity and autonomy needs.
  • Reliable multi-agent systems require:
    • Immutable state + versioning
    • Schema/data contracts
    • Circuit breakers
    • Compensation (sagas)
    • Strong observability
  • This is “unsexy infrastructure work” but makes systems dependable in production rather than just demos.

Speakers / sources featured

  • Sandipan Bhaumik (also referred to as “Sandy”)

Original video