Video summary

Запустил нейронку на 235B на домашнем сервере. Без видеокарты

Main summary

Key takeaways

Technology

Tech overview / goal

  • The speaker runs large language models (30B → 235B parameters) on a home server using only the CPU (no effective GPU acceleration).
  • Core emphasis: whether very large MoE/LLM models can be practical on consumer/enthusiast hardware using RAM capacity + quantization, and how performance changes with model size and CPU settings.

Planned/testing approaches include:

  • Synthetic benchmarks and real “live prompt” tasks to compare expectations vs. reality.
  • Testing multiple quantized models (various GPT/GLM/Qn variants) and comparing them to a high-end GPU baseline.

Hardware setup (CPU/RAM-centric)

New virtualization server configuration

  • Dual-socket motherboard (Chinese) with 2× LG20113 sockets
  • 2× Intel Xeon E5-2680 v4 CPUs
    • Total: 28 cores / 56 threads
  • 128 GB RAM
    • Configured as 8×16 GB DDR4
  • Storage: runs from NVMe SSD, but the model is loaded into RAM for execution.

RAM details / locality constraints

  • RAM speed note:
    • Initially expected 2133 MHz
    • Found to run at 2400 MHz
  • Key architectural constraint:
    • NUMA / socket memory locality matters
      • RAM is effectively split per CPU socket.
      • Cross-socket access is slower.
    • Performance depends on whether the workload stays mostly within one socket or spans both.

Core theory used to justify CPU viability

Why CPU performance may be “better than expected”

  • MoE (Mixture of Experts) concept:
    • Only a small subset of “experts” is used per token.
    • This shifts the bottleneck: speed depends more on memory read bandwidth/latency, not just CPU compute.
  • Practical implication:
    • If a model doesn’t fit in GPU VRAM, moving parts to RAM typically kills speed, and the speaker argues the GPU can become underutilized in mixed-memory scenarios.

Quantization

  • Weights are compressed to lower precision (examples: Q4 vs Q2).
  • Q2 is used to force extremely large models (up to 235B) to fit in available RAM—at a significant cost to accuracy/quality.

Software tooling / runtime approach

  • OS: Ubuntu 20.04
  • Tooling:
    • Installs utilities like git
    • Builds the LLaMA/C++ backend (CPP) directly from source
  • Avoids wrapper mode:
    • Runs via CPP directly, not Ollama, to avoid losing important test settings.
  • Uses NUMA enabled to enforce correct socket/memory awareness.

Benchmark methodology

Metrics

  • Measures two main throughput metrics (tokens per second):
    • PP = Prompt Processing (handles the input request)
    • TG = Token Generation (produces output tokens)

Repetitions

  • Records the first launch for reference.
  • For calculations, uses averages from about ~3 measurements.

CPU testing variables

  1. Hyperthreading on vs off
    • Checks how logical threads affect PP/TG.
  2. Single socket vs dual socket
    • On the same machine, tests resource partitioning:
      • One socket / 14 threads
      • Two sockets / 28 threads, with variants for hyperthreading
  3. Speaker’s expectation:
    • Dual-socket may help PP more than TG, but results vary by model.

Results: large models on CPU (30B–120B+ range)

Top tier (~120B-class) findings

120B GPT-like OSS

  • With hyperthreading
    • PP ~ 90 tok/s
    • TG ~ 10.5 tok/s
  • Without/less hyperthreading
    • PP ~ 83 tok/s
    • TG ~ ~11 tok/s
  • Takeaway: HT mostly helps PP slightly; TG is similar.

122B Qn 3.5

  • Without HT:
    • PP ~ 40.5 tok/s
    • TG ~ ~4 tok/s
  • With HT:
    • PP ~ 44 tok/s
    • TG ~ still ~4 tok/s

106B GLM 4.5 “Air”

  • Without HT:
    • PP ~ 33 tok/s
    • TG ~ ~5+ tok/s
  • With HT:
    • PP ~ 34.5 tok/s
    • TG ~ similar

Explanation for performance differences by model

Even with similar parameter counts, performance differs due to “active parameters” (MoE-style idea):

  • Qn 3.5: ~10B active
  • GLM 4.5 Air: ~12B active
  • GPT OSS 120B: ~5B active

Conclusion: active parameter count can strongly affect generation speed.

Results: smaller models on CPU (~26B–35B)

Three models tested (each with different “active parameter” counts).

Observations

  • Dual socket yields a near ~2× improvement in PP.
  • TG (generation):
    • Improves modestly or not much,
    • Except one Quen-corder–like model showed >30% improvement.
  • Hyperthreading:
    • Often helps PP,
    • TG gains can be negligible or sometimes negative depending on the model.
  • Models with “active parameter similarity” may still differ a lot in TG (example mentioned: about ~20 vs ~9.5 tok/s).

Practical conclusions

  • Dual-socket helps prompt handling more reliably.
  • Generation performance is model-specific, so it must be tested individually.

CPU vs GPU comparison (RTX 3090 baseline)

GPU baseline

  • GPU used: RTX 3090 (24GB VRAM)

35B model (~H3.5 35B)

  • GPU:
    • PP ~ 3300 tok/s
    • TG ~ 130 tok/s
  • Speaker conclusion:
    • Generation is ~10× faster
    • Prompt processing is vastly higher.

235B model (CPU-only feasible; GPU not usable)

  • CPU test results:
    • Without HT:
      • PP ~ 21 tok/s
      • TG ~ 4 tok/s
    • With HT:
      • PP ~ 23 tok/s
      • TG ~ still ~4 tok/s (HT slightly harms TG again)

Key claim

  • Consumer GPUs can’t run 235B at usable quality/speed in this context.
  • In this setup, CPU + RAM + quantization is the only feasible path.

Live task tests (quality vs synthetic benchmarks)

Instead of only token-speed numbers, the speaker tests code generation for a classic game:

Task: build a Snake game in JavaScript

Four models tested with iterative prompts asking to fix errors and improve visuals.

Example outcomes

~30B “Quen-corder”
  • First pass:
    • ~4m21s total
    • generation ~16.7 tok/s
  • Second prompt fixes “portals don’t work”:
    • time increases (~6m18s)
    • generation ~11.7 tok/s
  • Visuals: “pleasing,” but still functional limitations (portal logic depended on prompt iteration).
~106B GLM 4.5 R
  • Slower generation:
    • ~5 → 4.6 tok/s
  • Bugs:
    • game starts with issues
    • restart doesn’t work
    • field positioning shifts
  • “Fix control/restart” improvements:
    • only partially resolved the problems.
~120B GPT OSS
  • Mechanical correctness:
    • works from the first try (movement, portals, restart)
  • Visual improvements:
    • minimal (colors, portal pulsation)
    • suggests limited creativity or need for more specific visual instructions.
~235B (Q2-quantized due to RAM fit)
  • Extremely slow:
    • ~1 tok/s early
    • final ~2.6 tok/s
  • First output:
    • unusable HTML tag closing mistakes
    • minimal visuals
  • After follow-up prompt:
    • visuals improved (segmented snake, better portal separation)
    • some touch controls appeared even when not requested
  • Takeaway:
    • Q2 quantization severely degrades usable quality
    • synthetic speeds didn’t reliably predict real code viability.

Nuances / why performance changes over a long session

  • Context growth / chat history
    • Each new prompt expands stored context, slowing PP/TG.
    • Larger context also affects memory buffer needs.
  • Not all RAM is available to model weights
    • Extra memory is reserved for context/buffering.
  • CPU upgrades don’t fix generation much
    • Speaker claims the generation bottleneck is RAM bandwidth (approx. ~120–135 GB/s cited).
  • Synthetic vs real mismatch
    • Synthetic tests often assume fixed token workloads (e.g., prompt 512, generate 128).
    • Real tasks can diverge substantially in both quality and effective speed.

Cost / practicality analysis

  • Estimated build cost for 128GB CPU/RAM assembly:
    • roughly 60k–70k rubles (dominant cost is RAM)
    • note: RAM price inflation depends on market/used pricing conditions.
  • Recommendation:
    • For better cost/performance and speed/quality, consider GPU-based setups
      • e.g., NVIDIA RTX 30/40 class with 16–24GB VRAM
  • Dual-socket / large-RAM practicality:
    • Speaker questions whether extremely large RAM configurations make sense unless you truly need voluminous models that require heavy quantization and long waiting times are acceptable.

“Why local models at all?” (high-level use cases)

Main reasons local inference makes sense:

  1. Privacy (avoid sending data/code to external GPT/cloud services)
  2. Offline / local infrastructure (AI-dependent systems not tied to internet/VPN)
  3. Many small repeated tasks (cheaper than ongoing API/subscription costs)
  4. Experimenting / learning / tinkering (speaker’s primary motivation)

Main speakers / sources (as referenced in the subtitles)

  • Primary speaker: the YouTube creator/author (first-person; no name shown in subtitles)
  • Tools / repositories referenced:
    • AM CP (cloned repository)
    • LLaMA / CPP backend (built from sources)
    • Ollama / Lama wrapper (mentioned as an alternative)
  • Hardware referenced:
    • Intel Xeon E5-2680 v4
    • RTX 3090
  • Models referenced (spoken):
    • GPT OSS
    • Qn 3.5
    • GLM 4.5 Air / 4.5 R
    • Quen/Quencorder
    • H3.5
    • 235B quantized variant (Q2)

Original video