I Thought 2 TB of RAM Was Overkill… My AI Agents Disagree

Summary

The video investigates whether large amounts of system RAM meaningfully benefit AI agents, rather than focusing only on GPU memory (VRAM).

Key Concepts and Findings

• Agents use system RAM for more than overflow. Even when a model runs on the GPU, agent orchestration, tool calls, growing conversation context, and cached state occupy system memory. If an agent’s conversation is pushed out of RAM, the system may need to rebuild that context, wasting time.
• RAM and VRAM serve different purposes. The presenter concludes that system RAM affects what a machine can run and how much agent work it can keep active, while VRAM affects how many workloads can run concurrently on the GPU—and how quickly they run.

Tests and Results

Initial RTX Pro 6000 Setup

With 128 GB of system RAM and two RTX Pro 6000 cards, the presenter increased the number of concurrent coding agents. Eight ran comfortably, but performance deteriorated at 16, with some failures. Swap remained at zero and CPU use was around 6%, suggesting system RAM—not swap or CPU—was the limiting factor.

Lower-Cost GPU Test

The presenter replaced the RTX cards with an ASUS Radeon AI PRO R9700 with 32 GB of VRAM. Running a 124-billion-parameter mixture-of-experts model, about 5.5 GB was on the GPU while roughly 90 GB was in system RAM. The model ran at about 38 tokens per second, demonstrating that a large model can remain usable even when much of it resides in system memory.

• The Radeon handled roughly eight agents at a time, while the RTX setup sustained more.
• The Radeon was also slower for a single agent: about 28–29 tokens per second versus 63 on the RTX in the comparison shown.
• These GPU comparisons are not direct, like-for-like benchmarks because the Radeon used a different software stack.

RAM Upgrade

The presenter replaced 128 GB with 256 GB of ECC DDR5-6400—eight 32 GB modules—without changing anything else.

• In the reported comparison, the 128 GB setup had 30 agent-context rebuilds; with 256 GB, there were none.
• All 12 agents completed, and a 16-agent run also completed.
• RAM use peaked at about 150 GB.
• For the same 12-agent workload, the 256 GB run finished in 59 minutes, compared with 85 minutes for the 128 GB run, where only two agents finished.

The extra RAM did not increase each agent’s token rate; it reduced delays and repeated work.

Benchmark and Guide Information

The presenter introduces Agent Headroom, a benchmark suite for agent workloads available on GitHub as a 0.1 release. Its tests are named Rung, Pace, Crowd, Tenants, and Recall. Viewers are invited to run the benchmark and contribute; detailed explanations are deferred to a possible later video.

The video also shows the physical RAM installation, but does not present it as a step-by-step setup guide.

Sponsored Segment

A brief ad describes ChatLLM Teams as a dashboard for accessing language, image, and video models, along with Abacus AI DeepAgent and a coding editor. Advertised features include generating presentations and building or deploying apps from prompts.

Main Speaker and Source

Alex Ziskind presents his own hardware tests and analysis. He also mentions Kingston as the supplier of the RAM used in the upgrade.