Agents Sucked At UI (Until Today)

Summary

Theo describes using agent “hill climbing” to improve T3 Code’s UI. Instead of relying on a single prompt or a designer to revise the output, he had coding agents repeatedly change the interface, generate screenshots, and ask several AI models to review and score each round.

Improving T3 Code’s UI

• T3 Code is a free, open-source agentic development environment. Theo says the project does not have a full-time UI/UX designer and acknowledges that its existing interface needed work.
• His initial experiment asked an agent to make the UI more appealing and keep iterating until a panel of reviewers rated it at least 9/10.
• The panel used models from different providers to review the work, provide feedback, and score each round. Reviewers were instructed to evaluate screenshots and, in later rounds, compare T3 Code against Claude Code and Codex.
• The first reviews gave the existing UI scores around 6/10. Revisions addressed details such as typography, font hierarchy, card and border radii, inline code styling, branch-name readability, contrast, and consistency across interface elements.
• Scores improved over successive rounds. Theo then set a more demanding target—9.5/10 or higher in every category—and continued in a separate work tree. He reports that reviewers eventually reached that target, though one model’s scores varied and the process required many additional iterations.
• Theo says the resulting design felt clearer and more readable, with better visual hierarchy, more consistent styling, improved light-mode contrast, and more purposeful use of color. He also notes that an earlier iteration had become too grayscale, so he explicitly asked for color to be restored.
• At the time of the video, the refreshed design was available as a toggle in T3 Code’s beta tab.

What the Experiment Suggests—and Its Limits

Theo presents the method as a way to turn subjective work into a repeatable feedback loop: give agents a goal, have reviewers inspect the result, then use their criticism to guide another round. For more objective tasks, he recommends building a programmatic evaluation that assigns a score, allowing an agent to optimize against a measurable test.

He also stresses that UI scores are subjective and that LLM reviews are nondeterministic. The same model may score the same design differently depending on the prompt, comparison set, or context. Scores for competing products rose during some comparisons too, even though their screenshots had not changed. Theo therefore treats the scores as useful feedback, not definitive proof of design quality.

The work involved roughly 17.6 hours of parallel agent runtime. Theo estimates the broader effort cost around $400, with about $250 attributed to the UI work shown before a later refinement run. He notes that subscription usage and estimated model prices make the figure approximate.

Applying Hill Climbing to T3 Code’s MCP

Theo also describes using an evaluation to improve T3 Code’s MCP tool descriptions—the information agents use to decide how to work with T3 Code’s tools. He collected examples of agents choosing the wrong actions, such as opening a new thread instead of spawning a sub-agent, and turned them into test cases.

• The evaluation exposed failures across tool-use scenarios.
• Theo had agents first improve their success rate, then shorten the descriptions while keeping the evaluation score high, reducing the context consumed by tool definitions.
• He reports reductions of 52% in one configuration and 78% with tool search enabled. These changes were in the nightly build and were expected to reach the stable version soon.
• Unlike visual review scores, this evaluation could be run against collected cases, providing a more repeatable way to check whether changes preserved behavior.

Sponsor Segment: Magic Patterns

Theo demonstrates Magic Patterns OS, a design sandbox that can work from screenshots or source code. He says it can generate multiple UI options as real code rather than just mockups, support iterative design changes, and pass the resulting work back to a coding agent. He uses it to explore possible T3 Code interface changes, including a more compact sidebar.

Reviews and Guides Mentioned

• UI review loop: Generate screenshots, ask multiple models to critique and score them, apply the feedback, and repeat—ideally comparing against relevant alternatives.
• Evaluation-driven optimization: For tasks with measurable outcomes, build a test or score and have agents improve it while checking for regressions. Theo’s MCP evaluation is the main example.
• Magic Patterns walkthrough: A brief sponsor demonstration of generating and iterating on coded UI concepts from screenshots or existing source.

Main Speakers and Sources

• Theo, the video’s presenter, explains the experiments, results, and T3 Code changes.
• AI review models—including Claude/Opus, Codex, Muse, and Gemini—provide the design reviews and scores discussed in the video. Theo describes Gemini’s contributions as intentionally used for comic relief in some rounds.
• Magic Patterns is the sponsor featured in a product demonstration.