Stop Managing AI Projects Like Traditional Software

Summary

Traditional software roadmaps are built around delivery dates, feature tickets, staffing, and launch coordination. These practices can work when the solution is known, but they are poorly suited to novel AI capabilities: teams often cannot reliably estimate delivery before testing key assumptions.

The recommended alternative is to make roadmaps prescriptive about what to learn and improve, rather than primarily promising feature delivery dates.

Frameworks and playbooks

• Replace feature commitments with experiment-driven planning.

  • Start with genuine questions about whether a capability can work, rather than turning an assumed solution directly into an engineering ticket.
  • Define a measurable outcome for each question, estimate its likely impact, and use those estimates alongside effort to prioritize experiments.
  • Treat early estimates as hypotheses informed by small “pre-experiments” and data analysis, not as arbitrary guesses.
  • Expect plans to change as results come in. A well-maintained experiment backlog helps teams pivot with more confidence.

• Use a capability funnel to break down AI performance.

  • Map the assumptions a system must satisfy, from component-level requirements through to whether it accomplishes the user’s job to be done.
  • For a natural-language-to-SQL capability, for example, test whether it generates valid SQL, runs without errors, returns relevant results, matches user intent, and ultimately solves the user’s problem.
  • Evaluate components individually so failures point to actionable next steps, then assess the assembled system to see how failures combine.
  • Account for dependencies: an upstream failure can make downstream stages impossible to pass.

• Use a failure funnel to guide experiments and engineering work.

  • Break a capability into sequential stages and measure where it fails. The funnel can reveal which assumptions or components need attention.
  • Prioritize upstream failures carefully: fixing an early-stage problem can improve the chances of success across every downstream stage.
  • Use error analysis to identify patterns and write targeted evaluations. One practical heuristic is to record the most upstream error for each failed case.
  • For branching or agentic workflows, simplify the system into representative paths that can be evaluated from start to finish.

• Analyze evaluation results for explanations, not just scores.

  • Look for data stories that explain why performance changed and what to do next. A higher aggregate score alone may not identify the source of improvement or failure.
  • Compare experiments using stage-level results or transition matrices, and commit to the hypothesis and metrics in advance.
  • Make evaluation outputs interpretable so product and technical teams can discuss trade-offs and choose follow-up work.

• Build reproducible evaluation data.

  • Human labeling may be necessary to establish expected answers and trace failures back through the funnel.
  • For systems with a human in the loop, evaluate whether the system gives the human good enough inputs to succeed—not only whether the human’s final action was successful.
  • Data analysis and truth-seeking matter. Teams do not necessarily need a formally credentialed data scientist, but they do need people comfortable investigating data and testing explanations.

Metrics and illustrative estimates

These examples show how to make experiment proposals concrete; they are not universal benchmarks:

• Relevant tables are currently retrieved for 73% of queries. A proposed approach might improve this by a single-digit percentage.
• Attributes are currently extracted correctly 40% of the time. A proposed change might add 10% accuracy across the dataset.
• A conversation-classification task has a 7% false-negative rate. A proposed change might reduce it by 1 percentage point.
• In a RAG example, generations fail more often when the relevant document is not centrally ranked. A preliminary experiment suggests reranking could place the relevant item first 60% of the time. The subtitle’s stated 15% comparison is unclear, so it should be treated cautiously.
• In an example failure funnel, one stage has 41 failures. The figure illustrates the importance of upstream failures; the subtitles do not specify its unit or denominator.

Examples and recommendations

• For natural-language-to-SQL, evaluate more than whether the query runs. A correct answer may need to be broken down by time period or business segment to solve the user’s actual problem.
• For PDF extraction, test coverage across pages, tables, images, and captions, then separately evaluate downstream tasks such as creating tags or events.
• Ask users clarifying questions when their requests leave important requirements unspecified. The speaker cites OpenAI’s Deep Research as an example of a product that often prompts users for missing context.
• In a human-assisted recommendation workflow at Stitch Fix, the algorithm’s candidates needed to be good enough for stylists to select from. Poor final outcomes were not necessarily a failure by the stylist.
• To introduce the approach in an organization, start with one new AI capability or a high-priority existing one. Map its assumptions, build a failure funnel, and use the results to show where engineering effort should go.
• Vendor discussions should not substitute for testing a solution against the company’s own problem and evaluation criteria.

Presenters and sources

• Brian, Head of AI at Theory Ventures; previously Head of AI at Hex and Head of Data at Weights & Biases.
• Hamel Husain (referred to as “HML” in the subtitles), in discussion with Brian.
• Examples discussed include Stitch Fix, OpenAI Deep Research, and LangChain. Brian also mentions an upcoming Data Council talk on the failure funnel.