Video summary

If we want them to do Knowledge Work, design them as Knowledge Agents — Benjamin Clavié, Mixedbread

Main summary

Key takeaways

Technology

Core thesis: agents should do knowledge work, not “coding work”

The speaker argues that modern AI agents should be designed as knowledge agents/knowledge workers, analogous to how humans perform legal, medical, academic, actuarial, and research tasks.

He distinguishes between:

  • Coding agents: the knowledge is code.
  • Knowledge agents: the knowledge is information with ambiguity and context dependence.

What “knowledge work” means (two definitions)

  1. Information-input definition

    • Input is information (not physical materials).
    • Processing information is ambiguous/diffuse.
    • Output is actionable judgment/decisions (e.g., legal findings, medical recommendations).

  2. Search/knowledge problem definition

    • If the task requires search, it’s a knowledge problem.
    • Therefore, it qualifies as knowledge work.
    • Most service-economy work (e.g., law, research, academia, software engineering) falls into this category.

Why coding-focused agent design doesn’t generalize

Even though code is knowledge, it’s a special form:

  • It has durable cues (identifiers, file paths, references).
  • It is often more static and “grep-able.”

In non-code domains (e.g., law, contracts, medicine):

  • Meaning is implicit, contextual, and conditional.
  • The same phrase/term can map to different interpretations depending on domain and surrounding facts.

Implication: treating these tasks like “programming,” or building workflows around code-like assumptions, is insufficient.

“Loops” in knowledge work: tooling loop + organizational loop

The talk frames knowledge work evolution as an endless self-optimizing loop:

  • Tooling loop

    • New knowledge → better tools → new workflows/roles → more knowledge.

  • Organization loop

    • Progresses from single experts (“polymaths”) to libraries → universities → specialized bureaucracies/firms (e.g., hospital roles).

A key point is that tooling is not neutral:

Without the right tools, certain tasks may be not scalable or not cost-effective.

Better tools can make tasks cheap enough to scale (e.g., search engines replacing manual library navigation).

Benchmarks used: evaluating retrieval/search tools for deep research

The speaker describes evaluation settings used to quantify “tool quality”:

1. BrowseComp+Leaderboard (retrieval on a large corpus)

  • Scenario: deep research queries over ~200,000 documents.
  • Observations:
    • Poorly optimized lexical search baselines (e.g., BM25 variants) can be ~60% accuracy and effectively unreliable.
    • Optimization and better hybrids can raise performance to about ~98% accuracy.
    • Efficiency matters: achieving strong accuracy with fewer tool calls reduces tokens/resources, which is treated as “free cash.”
  • Main message: tool optimization can make tasks feasible to run repeatedly.
    • Too many tool calls = not usable in practice.

2. MQA (PDF-based enterprise QA / document understanding)

  • Mentioned in a joint-release context (Hugging Face + Snowflake).
  • Highlights interplay between:
    • Tools and orchestration/organization
  • Notes:
    • Even when BM25 is optimized, performance can plateau around what humans reach, implying tool limits still exist.
    • Human “unlimited search” still doesn’t guarantee retrieving the correct info if retrieval is insufficient.

Multi-modal retrieval and agent architecture improvements (Mixbride / Mixbragent)

The speaker credits gains from moving beyond OCR/text-only:

  • Better models + vision
  • Direct PDF/table reading
  • A multimodal search tool (“Gina” / “Mixbride search tool”) that can read PDFs and tables, producing large accuracy gains versus OCR-based baselines.

However, there remains a gap versus humans (speaker cites about 88.9 vs ~99.4 on the benchmark).

Proposed solution: Mixbr agent (agentic decomposition + sub-searchers)

  • An orchestrator/main agent decomposes the question into sub-aspects.
  • It writes targeted queries and dispatches searcher sub-agents.
  • Sub-agents produce small memos.
  • The main agent synthesizes the final answer from those memos.

Reported gains:

  • Accuracy improves by about +3.5 points.
  • Framed as reducing the “oracle gap” by roughly ~40% (fewer mistakes thanks to better architecture).

Practical “design rules” emphasized

  • Don’t overfit to tools

    • Tools aren’t magic; they help overcome ceilings.

  • Co-design tools with agents

    • Agents must know which retrieval primitive to use:
      • lexical (e.g., BM25) vs semantic retrieval over PDFs/tables
    • Otherwise agents may generate ineffective queries (e.g., grep/BM25-style queries when semantic retrieval is needed).

  • Orchestration matters

    • Context is expensive and finitely useful.
    • You still need to decompose tasks and orchestrate research rather than relying on a single huge context window.
    • Requires:
      • main agents + structure for organizing knowledge.

Key speakers/sources

  • Ben Clavié (Benjamin Clavié) — speaker; worked at Mixedbread (retrieval)

Mentioned/related systems and benchmarks:

  • BrowseComp+Leaderboard
  • MQA (Hugging Face + Snowflake)
  • Gina / Mixbride search tool
  • Mixbr agent (decomposition/orchestration approach)

Original video