If This Video Disappears, Remember What I Said About AI Companies

Summary

The video argues that a major risk of relying on AI products is not simply that models give imperfect answers, but that their behavior can change without users or developers being clearly informed. The host connects user complaints, research findings, provider documentation, and commercial incentives to make the case for stronger transparency and consumer protections.

Reported changes and evidence

• The host describes personal experiences and online reports of Claude and ChatGPT forgetting instructions, losing context, and requiring more supervision. They stress that these reports are not controlled experiments, and that some users say they have not noticed problems.
• A 2024 study by researchers from Stanford and UC Berkeley, published in Harvard Data Science Review, found that ChatGPT’s behavior changed over time. In one formatting task, instruction-following reportedly fell from over 99% to less than 1%, potentially breaking automated workflows.
• The video also cites an independent benchmark called “Live Nerf” in the subtitles. As of November 3, it had not found statistically significant deterioration in its tests of Opus 5.5. The host notes that its math and knowledge questions may not capture problems in long conversations, creative work, or complex coding.
• According to an Anthropic technical investigation discussed in the video, three Claude Code changes degraded performance without changing the underlying model:
  • Lowering the default reasoning effort from high to medium
  • A bug that discarded prior reasoning after an inactive period
  • An instruction to make responses shorter that hurt coding quality

Anthropic reportedly reversed some changes after complaints.

• The host also cites Anthropic’s acknowledgement of earlier infrastructure errors that affected only some requests, which may help explain why users experienced different performance.

Developers’ reliability problem

Pinned model IDs can protect developers from changes to a model’s weights and configuration, and providers publish model decommissioning schedules. But the video says this does not freeze the surrounding infrastructure—such as request routing, security checks, and response-generation systems—which may also affect outputs.

Other points raised include:

• A 2024 study by researchers from Carnegie Mellon University and HKUST reportedly found that about 59% of tested prompt-and-model combinations lost accuracy after API updates.
• Replacing a discontinued model can require substantial retesting and changes to prompts, integrations, error handling, and quality controls.
• The host argues that AI performance should be judged not only by how well a model completes a task, but also by the work and expense required to keep systems using it reliable.
• A 2025 paper from Oxford and the Alan Turing Institute is cited as arguing that “language models as a service” can undermine reproducibility, reliability, and trust.

Business incentives and regulation

The video suggests—while explicitly presenting this as analysis rather than proof of deliberate misconduct—that deep integration creates switching costs. Customers may be less likely to leave when replacing an AI provider means rebuilding and retesting workflows. Ongoing migrations and regression testing can also generate further demand for AI services.

The host argues that dramatic public debate about AI’s promise or danger can crowd out practical questions about what customers are owed when a paid service changes. They call for transparency measures such as:

• Notice of material performance or configuration changes
• Public version histories and meaningful documentation
• Information to help developers investigate regressions
• Clearer migration support when models are retired

The video cites existing consumer-protection rules, a 2025 paper by Serena Booth on applying consumer law to AI, a 2026 study on AI disclosure and user trust, a proposal for independent AI audits, and research on corporate influence over AI regulation. The host says these measures need not expose model weights or prevent providers from improving their products.

Product feature and guide highlighted

The sponsor segment demonstrates Lovable, a software-development tool that creates applications from plain-language descriptions. The host says they used it to build a mock-exam portal with questions, answers, explanations, and an examiner dashboard showing grades and areas of difficulty.

The mock exam is described as having 10 questions, four answer choices each, and some Latin-language surprises. A $5 discount through the host’s link is also mentioned.

Main speaker and sources

• Main speaker: The host of House of El: AI, speaking in the first person.
• Sources discussed: Reddit and OpenAI developer-forum reports; Anthropic’s product documentation and technical investigations; research from Stanford/UC Berkeley, Carnegie Mellon/HKUST, Oxford/the Alan Turing Institute, and other academic groups; and a class-action complaint filed on behalf of Claude paid subscribers. The video notes that the lawsuit’s allegations had not been proven in court.