Inference Will Be $10TRN Market: What No One Sees | Software Investors are F***ed | Naveen Rao

Energy and AI Infrastructure

Rao argues that energy availability will become a major constraint on AI expansion within a few years. The US has relied on spare grid capacity, but that buffer is likely to shrink as data-center demand grows.

He contrasts the difficulty of building power infrastructure with the scale of investment required. In his view, adding around 10 GW of capacity per year would be an enormous undertaking, while adding 40 GW annually is unrealistic under current conditions. He also says China has expanded power capacity much faster than the US over the past two decades.

Data centers may pay more for dedicated or “behind-the-meter” power than for ordinary grid electricity. Rao mentions Bloom Energy’s proposed projects as a possible example, while noting that such approaches still need to prove themselves in practice.

He sees current concerns about AI infrastructure demand as more likely a temporary supply-and-deployment challenge than a burst bubble. His case depends on continued growth in demand for AI services.

Improving Compute Efficiency

Rao describes three broad levels of efficiency improvement:

1. Physical scaling: Making devices smaller and more efficient. He believes major gains at this level have slowed.
2. Compute architecture and data formats: Using lower-precision formats, such as 16-, 8-, and 4-bit computation.
3. Data locality and reduced movement: Minimizing how much information must move between parts of a system.

He argues that future gains will increasingly come from the latter two levels, especially by designing models, computing architectures, and physical hardware together. Rather than treating a model as fixed and optimizing hardware only for speed, he proposes prioritizing energy efficiency and the amount of intelligence produced per unit of energy.

Specialized Chips and the Value of Hardware

Rao expects computing hardware to become more specialized for particular workloads, rather than converging on one universally optimal chip. He sees this as a continuation of the broader trend toward workload-specific architectures.

He does not expect AI chips or the broader hardware stack to become ordinary commodities soon. His reasoning is that demand for intelligence is effectively unbounded, and hardware providers can create value by converting energy into useful computation more efficiently.

He credits Nvidia with anticipating the importance of AI early, while arguing that hardware remains a promising area for investment over the next decade.

AI Creation, Verification, and Trust

AI makes it faster and cheaper to produce code, models, and other work products. Rao says this shifts an important bottleneck from creating outputs to verifying that they are correct and meet a specification.

Verification involves more than a quick check: it requires defining what a good result is, testing outputs against that specification, and establishing confidence in the result.

He cautions against trusting AI agents as much as experienced people today. Expertise depends on seeing many cases and developing judgment; AI systems can generate plausible but incorrect answers, and long chains of generated reasoning can compound errors. Nevertheless, he predicts that people may eventually trust agents more than themselves as systems become more capable and their training and verification improve.

Data, Training, and Model Architectures

Rao expects data and training methods to remain important and increasingly specialized, rather than becoming generic commodities. High-quality datasets and methods for selecting and using them can provide an advantage.

He describes a shift toward post-training methods, including reinforcement learning (RL). RL can be useful when a task has a verifiable reward—for example, checking whether an action achieved a measurable result. He says RL may be less efficient in improvement per unit of compute or cost, but can still be important for making models perform valuable tasks.

Rao does not believe transformers are the final architecture. He argues that today’s approach to reasoning—generating and comparing possible traces—may be inefficient, and expects future systems to use more effective ways of reasoning.

Commoditization, Software, and Jobs

Rao expects some model capabilities to become commoditized: older or open models may remain useful for routine tasks, while the most advanced models retain value for demanding work.

He predicts that inference prices will fall and provider margins may shrink, but distinguishes this from hardware, where efficiency improvements may continue to support competitive advantages. He estimates that open models could handle about 80% of token traffic within three years, while closed models could still account for a larger share of spending on high-value tasks. These are his projections, not established outcomes.

He is skeptical that traditional SaaS will remain as large a profit opportunity over the long term. He expects more companies to combine AI software with human review to deliver outcomes—for example, handling legal or engineering work faster and more cheaply.

Rao does not expect AI necessarily to eliminate developer jobs. If development becomes cheaper, organizations may demand more fixes, features, and release cycles. Skilled people who can recognize quality and verify AI-generated work may become more valuable.

Robotics and Investment Outlook

Rao sees robotics as a much larger long-term opportunity than coding agents, arguing that AI in the physical world could help solve a broad range of practical problems. He suggests working prototypes for some tasks could emerge in two to three years, with a market developing over five to seven years; the host is more cautious about timing.

Rao expects the model and robot to be closely integrated, rather than treated as separate products. He says his interest is mainly in the technology stack and model companies, not yet in consumer applications.

For deep-tech investing, he argues that technical progress may be a better early-stage measure than market validation when the problem and potential demand are already clear. He also says investors need to assess the team’s ability to navigate uncertainty and make technical pivots.

He says he would consider building purpose-designed manufacturing capacity to avoid adapting mass-market hardware to specialized needs.

Rao identifies AI customer-support tools and agent-programming tools as crowded or less differentiated investment areas, arguing that model providers may capture more of the value.

Reviews, Guides, and Tutorials

No product reviews, step-by-step guides, or tutorials are presented. The video is an interview focused on analysis and forecasts.

Main Speakers and Source

• Naveen Rao — Technology entrepreneur and founder of Unconventional AI; discusses AI hardware, energy, compute, software, robotics, and investing.
• Harry Stebbings — Host and interviewer; asks questions about infrastructure, markets, AI’s effects on software and labor, and investment strategy.
• Source: 20VC interview with Naveen Rao.