The Sol 6.1 Benchmarks Are STUPID, So I Tested It vs Sonnet 5.5

Summary

The creator compares OpenAI’s GPT 6.1 “Sol” with Anthropic’s Sonnet 5.5, arguing that benchmark scores alone don’t show how the models perform on real tasks. The key trade-off is that Sol is much more token-efficient and cheaper, while Sonnet is generally faster and sometimes produces stronger results.

Benchmarks and cost

• Results cited in the video are mixed: Sonnet leads slightly on Artificial Analysis indices and by a wider margin on Automation Bench, while Sol leads on DeepSuite.
• Although both models are listed at the same per-token rates, the creator says Sol often uses far fewer tokens. In one benchmark comparison, both scored 52%, but Sol’s cost per task was reported to be about one-quarter of Sonnet’s. Some benchmark tasks reportedly showed an even larger cost gap.
• Sonnet was reported to be about twice as fast overall. The creator cautions that benchmark figures don’t necessarily predict results on practical work.

Real-world tests

1. JavaScript explainer video

   • Both models were asked to create a 15-second vertical explainer about Claude’s use of subagents.
   • Sol’s visuals were acceptable, but its soundtrack felt out of sync with the content.
   • Sonnet produced a more cohesive sequence, with audio that better matched the visuals.
   • Token use: about 70,000 for Sol versus 300,000 for Sonnet.

2. Boutique hotel landing page

   • Both built a site for a fictional hotel called Dune House.
   • Sol produced a clean, professional design, but its booking section lacked a prominent way to check availability.
   • Sonnet’s availability controls were more functional, though the creator criticized some design choices, including the fonts. The overall design preference was subjective, and the creator called the result a draw.
   • Token use: about 109,000 for Sol versus 200,000 for Sonnet.
   • Sonnet lacks built-in image generation in this comparison, so it was allowed to use the Higgsfield API. The video notes that Higgsfield offers pay-per-use access to its image and video models.

3. 3D travel dashboard

   • Sol made an interactive globe with routes, weather, and destination details, but the creator found it relatively limited in substance and noted a blinking black rectangle.
   • Sonnet’s version was judged more visually impressive and included destination images and additional city information, although the creator said it was not especially functional.
   • Sonnet was preferred for this visual-focused task.
   • Token use: about 150,000 for Sol versus 400,000 for Sonnet.

4. Browser tank game

   • Both models were asked to make a browser-based tank game inspired by World of Tanks, using Three.js.
   • Sol offered tank choices, ammunition types, a live map, and team health indicators. Sonnet provided more tank statistics and a larger-feeling map, but the creator found its tanks’ atmosphere and appearance less convincing.
   • The creator considered the results roughly equal overall, with preferences depending on controls, visuals, and interface.
   • Sol took over two hours and used roughly half as many tokens as Sonnet; Sonnet finished in about an hour and used around 700,000 tokens.

Overall conclusion

The creator sees Sol as a strong, inexpensive option for many real-world tasks, especially when token efficiency matters. Sonnet 5.5 tends to be faster and won or tied some of the creative comparisons, but used substantially more tokens. The recommendation is to choose based on the task, budget, speed requirements, and preferred output—not benchmark scores alone.

The video also discusses OpenAI reducing the usage multiplier on its $200 plan from 20× to 10×. The creator compares this with Anthropic’s 20× plan, noting that the two companies’ usage limits are defined differently and should not be compared without considering those terms.

Main speaker/source

• Chase AI — the video’s creator and reviewer.