The 5 Levels of AI Coding (Why Most of You Won't Make It Past Level 2)

Summary

The video argues that AI coding is producing two very different outcomes: a small number of teams are building production software with autonomous AI agents, while many developers using AI in conventional workflows are initially becoming slower. The speaker’s central claim is that the gap is less about access to better tools than about redesigning workflows, organizations, and skills around AI.

The six levels of AI-assisted coding

Drawing on a framework attributed to Glowforge CEO Dan Shapiro, the speaker describes increasing levels of AI autonomy:

1. Level 0 — “Spicy autocomplete”: AI suggests the next line; the human writes the software.
2. Level 1 — Coding intern: AI completes a discrete, well-scoped task, and a human reviews its work.
3. Level 2 — Junior developer: AI makes changes across files and modules, while the human still reads the code. The speaker estimates that many developers who consider themselves AI-native are at this level.
4. Level 3 — Developer as manager: The human directs AI to implement features and reviews work at the feature or pull-request level. The speaker says many developers plateau here, in part because they find it difficult to relinquish close control of the code.
5. Level 4 — Developer as product manager: The human writes a detailed specification, then evaluates whether the delivered software works, without routinely inspecting the code.
6. Level 5 — “Dark factory”: Specifications go in and working software comes out; agents write, test, and ship without human code-writing or code review.

The speaker says vendors’ claims about “AI writing code” can refer to very different levels of autonomy, so it is important to distinguish marketing language from how a system operates in practice.

How the “dark factory” works

StrongDM is presented as a production example of Level 5: a three-person team whose software factory is organized around specifications and outcome evaluation, rather than conventional coding and coordination practices.

• Markdown specifications describe the desired software. Agents use them to build and test the system.
• External behavioral scenarios evaluate the result but are kept outside the codebase and hidden from the agents during development. The speaker compares these to machine-learning holdout sets: they make it harder for an agent to optimize for visible tests instead of meeting the intended behavior.
• Digital twins simulate external services such as Jira, Slack, and Google services. Agents can run integration scenarios against these replicas instead of real production systems, APIs, or data.
• The speaker says StrongDM’s factory has produced real production software, including its CXDB context store. He also reports the team’s rule of thumb that a factory spending under $1,000 per human engineer per day may have room to improve.

The larger point is that autonomous coding depends on more than an agent: it requires reliable evaluation, a sufficiently detailed specification, and an environment in which software can be tested safely.

Why many developers see productivity fall

The speaker contrasts reports of AI-generated code at Anthropic and OpenAI with a METR randomized trial, which he says found experienced open-source developers using AI tools took 19% longer on tasks. He also says participants believed they had become faster, illustrating a gap between perceived and measured productivity.

The explanation offered is workflow disruption. Developers may spend time checking suggestions, correcting nearly-right code, switching between their own reasoning and the model’s, and diagnosing subtle generated-code errors. Adding an AI assistant to an unchanged process can therefore create a productivity dip—the bottom of a “J-curve”—before improvements appear.

The speaker argues that organizations with meaningful gains redesign the development process end to end, rather than simply installing a tool or running a brief training session.

Organizational and technical changes

Many familiar software processes exist to coordinate human work and compensate for human limitations. In a highly autonomous system, practices such as sprints, stand-ups, manual QA, and conventional code review may need to change or serve different purposes.

The speaker says management work is shifting from coordinating implementation to:

• Defining precise specifications and success criteria.
• Building pipelines of work that agents can execute.
• Evaluating outcomes and ensuring they serve users.
• Updating CI/CD, testing, review, and deployment processes for AI-generated code.

The bottleneck, in this account, is moving from implementation speed to specification quality. Agents cannot reliably fill gaps in a vague request using human context; they may instead make guesses. Strong specifications require deep knowledge of the system, customers, and business problem.

A migration path for existing systems

Because most enterprise software is existing, or “brownfield,” the speaker cautions that organizations cannot simply deploy agents and expect autonomous development. Legacy systems often contain undocumented behavior, incomplete test coverage, and institutional knowledge that exists mainly in employees’ heads.

The video proposes a gradual transition:

1. Use AI at Levels 2–3 to assist with existing development tasks.
2. Document what the current system actually does and build scenarios that capture its real behavior.
3. Adapt CI/CD, testing, review, and deployment processes for AI-generated work.
4. Move suitable new development toward Level 4–5 patterns while maintaining legacy systems in parallel.

The speaker emphasizes that this transition may take months or years, depending on the organization and its willingness to undertake difficult process changes.

Workforce, skills, and the software market

The speaker warns that AI automation could weaken the traditional junior-engineer apprenticeship path, since entry-level tasks such as simple features and small bug fixes may be automated. At the same time, he argues that software still needs strong engineers—especially people with systems thinking, customer understanding, product judgment, and the ability to specify and evaluate work.

He predicts greater value for broad problem-solving ability and AI fluency, rather than expertise limited to a narrow technical stack. He also describes simulated, AI-assisted training environments as one possible way to develop judgment, while noting that this is not the same as learning by writing software from scratch.

Finally, the speaker argues that cheaper software production could expand the market: organizations that could not previously afford custom software may become able to build it. In his view, lower production costs shift the key constraint toward deciding what should be built and for whom. He stresses, however, that this potential growth does not remove the immediate disruption to jobs and career pathways.

Guides and practical resources mentioned

• Migration roadmap: The video outlines the four-step progression above for moving from AI assistance toward more autonomous development.
• Further exercises and resources: The speaker says he has published additional exercises and resources on his Substack. The video itself is analysis and a framework, rather than a hands-on coding tutorial.

Main speaker and sources discussed

• Main speaker: Nate B. Jones.
• Framework discussed: Dan Shapiro’s five-level “vibe coding” framework.
• Production example: StrongDM’s software factory and its team.
• Other sources referenced: Simon Willison’s assessment of StrongDM; the METR randomized study; claims and examples attributed to Anthropic and OpenAI; and workforce or adoption figures attributed to Harvard research and Gartner.