Video Summary - Opus 5.5 vs GPT-6 Sol: Tested on 50,000 Lines of Code

Key test setup & performance results

Real project scale: ~50,000 lines of code across ~500 source files.
Documentation provided: an extensive spec/design document (about 20+ pages) describing the intended solution.

GPT-6 “Sol”: ~15 minutes, about $5
Opus 5.5: ~50 minutes, about $50
Additional note (Fable 5.1): ~50 minutes, about 2× Opus 5.5, and it found similar problems.

The project involves AI agents that must select the correct internal “domain” (e.g., Sales, Marketing, HR) to answer questions and access data.
The data model assumed a strict separation between departments/domains.
The author emphasized real-world scenarios where answers require cross-domain data, such as joining sales + marketing tables.
The documentation review aimed to uncover critical architectural/data-modeling risks that could cause future failures, not just minor bugs.

Cross-domain entity/metric/table linkage was not properly constrained
- Business entities included definitions, metrics, and tables (and terms that can span domains).
- Example issue: a term like “ARR / revenue” may apply to Sales and Marketing, but the associated metric/table might be intended to be Sales-only.
- Risk: an agent answering a Marketing question might incorrectly access a Sales-only table due to how connections/domains were modeled.
- Conclusion: connections should be blocked or constrained when the domain context doesn’t match.
Table views inherited/propagated description/metadata decisions incorrectly
- A table view duplicates a base table so changes don’t affect the original.
- The issue: view description behavior wasn’t fully specified—for example, if the view has an initial description and the source table’s description changes, it’s unclear whether the view should update.
- GPT-6 Sol noted this decision wasn’t recorded in the project document.

Risk of deploying an old application version via build/deploy flow
- During database + service migration, a build artifact (e.g., a Docker image) is created.
- Identified problem: deployment could pull/push an old version from a different Git branch, causing stale code in production.
Rollback procedure documentation gaps could cause duplicate identifiers
- During migration rollback, Opus found the spec didn’t mention cleaning documents before restoring backups.
- Risk: restoring backups without cleanup could lead to duplicate identifiers, which would cause downstream database issues.

The author warns that finding more defects does not automatically mean a better model.
The emphasis is on discovering serious, future-impacting architectural risks, not minor imperfections.
Both models were considered strong at finding issues the other missed.

The author asked whether the plan could be fully implemented by AI agents using documentation alone.
Model guidance (as summarized):
- They can do most of it, but still need:
  - key decisions
  - access/permissions
  - independent verification
Both models allegedly provided guidance on steps toward making the process more autonomous.

Both ran on Ultra Code (described as high-level logic plus supporting agents).

Best practice: run both models independently and combine results (optionally use a third model for synthesis).
Model selection guidance:
- For highly ambiguous early-stage documentation/spec review: lean toward Opus 5.5
- For coding efficiency/cost: prefer GPT-6 Sol
- For very complex large-scale corporate systems: GPT-6 Sol may cover most needs (per the author)
If choosing one: the author suggests GPT-6 Sol (“Codeex”).
Mentions “Luna” as cheap and recommends using it more often (as a contextual/secondary suggestion).

The video author/tester (narrator presenting and comparing Opus 5.5 vs GPT-6 Sol)
“Astra” (referenced as an orchestration/next-step reviewer: “Astra—start with Sol, and for high-risk releases, add Opus”)
“Opus 5.5” (model under test)
“GPT-6 Sol” (model under test)
Ultra Code (agent framework used to run the models)