SpaceX AI engineer, Lauren Tan

Summary

The talk focuses on building enough trust in coding agents to delegate more work safely. Lauren Tan argues that verification—not simply prompting—is the key capability: agents should run the application, inspect its actual behavior, and test their changes. Verification does not guarantee good design, she says, but it helps establish that code works and reduces the engineer’s role as the manual testing bottleneck.

Building and Maintaining Verification Skills

Tan describes creating agent “skills”—instructions and supporting tools that teach agents how to work in a specific codebase. For an app, a verification skill might let an agent use the Chrome DevTools Protocol, collect performance traces or heap snapshots, or control an iOS simulator.

A skill alone may not be enough: an agent also needs to understand how the product is organized and how to reach particular features. Tan’s feature map records information such as UI navigation, features, keyboard shortcuts, and DOM selectors. This helps an agent investigate vague bug reports or screenshots instead of guessing where to look.

Her Pstack plugin grew incrementally from skills designed to address observed agent failure modes. It includes tools to create and maintain verification skills. Tan recommends starting locally, where developers can observe how an agent interacts with an application, before relying on cloud agents.

Evaluating Agent Skills

Tan describes an eval as a kind of unit test for an agent skill. In Pstack’s eval playbook, a coordinator agent creates a rubric and launches subagents to test the skill. The evaluation setup is designed to avoid telling subagents that they are being evaluated, since that knowledge could affect their behavior.

Skills can be tested across different models, and a separate judge model can help check for bias in the scoring. Tan also uses repeated eval runs to improve a skill against its rubric.

She cautions that maintaining skills takes ongoing observation: developers should inspect tool calls and agent behavior, identify recurring failure modes, and turn those lessons into instructions or stronger safeguards.

Scaling from Local Work to Cloud Agents

Tan presents agent adoption as a gradual “trust curve.” She advises against jumping straight from closely supervising one or two agents to spawning large numbers of cloud agents; that can waste time and tokens. Once verification is reliable, cloud agents can take on work triggered by team workflows.

For example, she describes an internal agent called Benny that takes bug reports, launches a cloud environment, attempts to reproduce the issue, and reports what it finds. In one case, it confirmed that a reported bug was already fixed on the main branch.

Tan says that, in her own workflow, agents had reached the point of auto-merging some pull requests, which she reviewed afterward on the main branch. She presents this as the result of substantial setup and iteration, not as a recommended starting point.

Architecture and Safeguards for Agent-Written Code

Tan argues that codebases need strong constraints, especially when agents are working in them. Existing, well-structured applications may already have useful conventions and guardrails, while quickly built prototypes can lack them and accumulate inconsistent shortcuts. She describes refactoring Grokbot—with the help of agents—to establish a stricter architecture, which she says took hundreds of pull requests.

For Grokbot, she describes an internal architecture called Dune, designed for agent-powered Electron applications. Its practices include:

• Keeping feature code together in conventional, predictable directories.
• Enforcing boundaries between Electron main-process and renderer code.
• Using CI checks and static analysis to block unwanted patterns, including banning React’s useEffect in that codebase.
• Applying lints and other checks for patterns the team has found agents frequently get wrong.
• Using additional guidance, skills, and a code-review tool called Bugbot, while treating these as softer safeguards than rules enforced by CI.

Her general recommendation is to turn recurring code-review corrections into lint rules, CI failures, or architectural constraints where possible. She prefers small, focused pull requests because they are easier to understand and revert, but says the team does not impose a hard PR-size limit.

Costs, Productivity, and Broader Access

Tan acknowledges that setting up verification, refactoring a codebase, and running agents can consume substantial tokens. She frames that spending as an ROI decision: the investment may allow a small team to move faster and enable more people to contribute without increasing engineering overhead. She also notes that agents can make rigid architectural rules practical because they can absorb the extra work those rules impose.

She describes Grokbot as a more accessible agent interface for people outside engineering, with a familiar, messaging-style experience and agents that can be individually named and coordinated. In her account, product managers and designers use it to summarize work, report bugs, and contribute code, with the Dune architecture intended to keep those contributions sustainable.

Practical Advice

1. Start with local verification: Teach an agent to run the application and inspect real behavior before scaling to cloud workflows.
2. Build a feature map: Document how to navigate product features and identify relevant UI elements.
3. Evaluate skills systematically: Use rubrics, multiple subagents, cross-model testing, and repeated eval runs.
4. Scale gradually: Move to cloud agents and automation only after building confidence in the verification process.
5. Make safeguards enforceable: Prefer architecture, static analysis, and CI checks over relying only on prompts or documentation.
6. Keep changes reviewable: Split work into focused pull requests where practical, even without a strict size limit.

Main Speakers and Sources

• Lauren Tan: Main speaker; discusses her work with Cursor’s agents and the Grokbot codebase, and presents the verification, evaluation, and architecture practices.
• Colin: Host and interviewer; asks follow-up questions about Pstack, evaluations, cloud agents, pull requests, CI, token costs, and product-team usage.