Video summary

Why Your AI Engineering Projects Won’t Land You a Job (The 5 Levels of AI Engineering Projects)

Main summary

Key takeaways

Educational

Main ideas / lesson

  • The video argues that AI engineering “projects that get you hired” typically progress through five levels, each requiring qualitatively more engineering rigor:
    1. Basic LLM app interfaces (prompting + light structure)
    2. Reliable systems with RAG and component-level evaluation
    3. Agents that can take actions in multi-step loops
    4. Enterprise-scale multi-agent systems with operational reliability, tracing, guardrails, and cost/latency control
    5. Self-improving systems where agents help build and refine the system with minimal human intervention—requiring advanced, cheat-resistant evaluation and deep production observability
  • A recurring warning: people get stuck by building the wrong type of project (e.g., impressive demos that can’t be measured, validated, or scaled).
  • The speaker claims coaching experience: they’ve coached “hundreds of people” and seen many candidates stuck.

The five levels (concepts + what you build + how it changes)

Level 1: LLM API apps with a simple UI (demo-grade usefulness)

What you build

  • A project that calls a language model API behind a simple interface.
    • Examples:
      • A customer Q&A chatbot
      • A feature that summarizes articles

Typical implementation characteristics

  • Usually uses OpenAI or Anthropic APIs.
  • Often wraps the model with something like a Streamlit app.
  • You do prompt engineering, but often:
    • You don’t track or test prompts
    • You may have structured outputs
    • You might add tests to verify parsing
    • You may add fallback logic when the model “flakes out”

Key limitation

  • Outputs aren’t reliably measured:
    • No metrics to evaluate response quality
    • No systematic testing to confirm improvements actually help
    • The bot may miss important company-specific context because it relies on training + whatever it can find on the web

Goal to advance (from Level 1 to Level 2)

  • Add your own evaluation and connect the system to relevant unseen user/company data:
    • Example approach: take an existing chatbot and hook it to data the model hasn’t seen.

Level 2: RAG (Retrieval-Augmented Generation) + rigorous evaluation

What you build

  • A more robust system that can answer using internal documentation (or other private knowledge).

Core concept

  • Add RAG so the model can retrieve context from internal knowledge bases before answering.

What becomes “the hard part”

  • Ensuring the system gives correct answers, and diagnosing failures.

Engineering details emphasized

  • Document preparation + retrieval quality:
    • Chunking documents properly
    • Embedding them
    • Storing vectors in a vector database
    • Choosing methods for retrieving the right chunks for each new question
  • Specialized evaluation methods:
    • Evaluate each system component (not just the final text)
    • Evaluate the whole system
    • Determine what “good” looks like and how to measure whether responses satisfy the customer need

Goal to advance (toward Level 3)

  • Extend beyond question-answering so the AI can do something with retrieved information (i.e., take actions).

Level 3: AI Agents (multi-step tool use with decision loops)

What an agent is

  • A model that:
    • Uses tools
    • Makes decisions on its own
    • Operates in a loop:
      • decide what to do → act → check result → decide next step → repeat until done

Actions it can perform

  • Examples of tool actions:
    • Search the web
    • Call other APIs
    • Run code
    • Update a database

Examples given

  • A research agent that gathers info from many sources to help write a brief
  • A customer support agent that looks up orders and processes requests (even via phone)

Why this level is “dangerous”

  • Mistakes compound:
    • Level 2 evaluation scores answers
    • Level 3 evaluation must grade many decisions across steps
    • One wrong turn can invalidate everything that follows
  • Real consequences are possible:
    • credit card mistakes
    • code mistakes
    • email mistakes

Goal to advance (from Level 2 to Level 3)

  • Convert your Level 2 system into an agent by:
    • giving it a real toolset
    • starting with tools that are read-only (safe mode)
    • then enabling more capabilities only after reliability/security is verified

Safety/robustness emphasis

  • Add:
    • observability/instrumentation
    • confidence in agent reliability
    • security controls before allowing high-impact actions (e.g., ordering groceries)

Level 4: Enterprise-scale multi-agent systems (production reliability at scale)

What changes

  • Instead of one agent doing everything, you build a team of specialized agents.
  • Agents hand work off to each other and may include:
    • planners/routers
    • specialists
    • checkers/reviewers

Practice example

  • Customer support:
    • one agent routes/reviews the ticket
    • billing and technical agents handle parts
    • a final agent reviews the response before the customer sees it

Scale requirement

  • Must work for millions of concurrent users.

What becomes the “real work”

  • Coordination + operations:
    • Each added agent is another failure point
    • Mistakes propagate down the chain and become public/expensive

Reliability/scaffolding beyond earlier levels

  • The video claims that for systems like Claude Code, most of the code isn’t the model itself but the scaffolding.
  • Adds operational capabilities such as:
    • tracing (see exactly what each agent did when things go wrong)
    • guardrails (prevent data leakage and prompt injection-style attacks)
    • controlling cost and latency

Goal to advance (from Level 3 to Level 4)

  • Even if you can’t reach “millions” personally, you can design a scalable architecture:
    • split tasks across multiple agents
    • include a checker agent
    • intentionally test adversarial cases (e.g., feed fake ingredients to see if the system catches errors)
  • Use cloud services and cost/reliability thinking from the beginning.

Level 5: Self-improving systems (agents that improve the system itself)

What changes at this level

  • You “get out of the way”:
    • advanced systems allow agents to improve themselves with minimal human direction
  • This is beyond:
    • coding agents writing code (which occurs at every level)
    • spec-driven development (agent implements from requirements)

Self-improvement cycle described

  • An agent/system loop that:
    1. checks how the system is currently performing
    2. proposes a change to improve it
    3. implements the change
    4. tests whether it helped
    5. keeps it or discards it
    6. repeats for hours/days without a human

Evidence/examples cited

  • Andrej Karpathy: a single agent ran ~700 experiments over two days, returning ~20 improvements for faster model training.
  • Anthropic: “most of the code getting merged” was written by Claude.

Why this is extremely hard

  • The cycle depends on the agent being able to judge its own work reliably.
  • If scoring is slightly wrong:
    • the agent can exploit the evaluation
    • achieve high scores while degrading the real product

Advanced skills required

  • Design evaluation systems that agents can’t cheat:
    • performance must correlate with what makes the product genuinely good
    • prevents manipulation of easy metrics
  • Deep instrumentation/observability:
    • you can’t watch the system live
    • you rely on the trail/logs it leaves behind
  • Define where humans still must be involved:
    • humans typically pick the problem, design scoring, and approve production changes

Positioning / takeaway

  • The speaker says most people don’t need Level 5.
  • Their own production work at Twitch is often Level 2 or 3, and sometimes a “well-done prompt + simple evaluation” is enough.
  • They emphasize being “useful, not fancy.”

Methodology / “how to progress” checklist (implicit instructions extracted from the video)

  • Start at Level 1

    • Build a small LLM app with an API + simple interface (e.g., chatbot or summarizer)
    • Use structured outputs and basic parsing tests
    • Add fallback logic for model failures (if needed)

  • Move to Level 2

    • Connect the app to relevant data the model hasn’t already seen (e.g., internal docs, user-specific logs/recipes)
    • Implement RAG:
      • chunk documents
      • embed chunks
      • store in a vector DB
      • implement retrieval for relevant chunks
    • Create evaluation:
      • metrics that reflect the customer need
      • test component-by-component
      • test the system end-to-end

  • Move to Level 3

    • Convert question answering into tool-using agents:
      • give a toolset (start read-only)
      • implement the action → check → next-step loop
    • Add observability and security checks before enabling high-impact actions

  • Move to Level 4

    • Build a multi-agent team:
      • specialist agents + routing/planning
      • checker/reviewer agent before user-visible output
    • Add enterprise operations:
      • tracing
      • guardrails against data leakage
      • cost + latency control
    • Test failures intentionally (adversarial inputs, validation bypass attempts)

  • Move to Level 5 (only if truly aiming for cutting-edge work)

    • Implement a self-improvement loop with robust “can’t-cheat” evaluation
    • Use heavy instrumentation to reconstruct behavior after the fact
    • Keep humans in the loop for problem definition, scoring design, and production approval

Speakers / sources featured (identified)

  • Primary speaker/creator: “I’m mapping out…” / “Today, I’m mapping…” (video narrator/coach; name not provided in the subtitles)
  • DataCamp (course/program sponsor and referenced curriculum)
    • “Associate AI Engineer for Developers” track
    • “Associate AI Engineer for Developers” certification (theory exams + practical)
  • Andrej Karpathy (cited for an experiments example involving ~700 experiments)
  • Anthropic / Claude (referenced as Claude code behavior and that Claude writes most merged code)
  • OpenAI (mentioned as an API option)
  • Anthropic (mentioned as an API option and platform)
  • Hugging Face (mentioned as a tool for embeddings/model ecosystem)
  • LangChain (mentioned as a tool for building LLM apps)
  • Pinecone (mentioned as a vector database/embeddings tool)
  • Streamlit (mentioned as an interface used in Level 1 apps)
  • Twitch (speaker’s employer experience mentioned)

Original video