Let’s Handle 1 Million Requests per Second, It’s Scarier Than You Think!

High-level overview

• Purpose: simulate and analyze handling ≥1,000,000 HTTP requests per second (RPS). The creator builds, benchmarks and iterates on an end-to-end stack (client load generators → web servers → Redis / Postgres) to expose real-world bottlenecks, costs and design patterns.
• Key message:

  At extreme scale you must optimize algorithms, architecture and tooling — small mistakes (O(n) vs O(log n), wrong storage choice, wrong language/framework) become catastrophically expensive. Monitoring, profiling and trade-offs (cost vs latency vs durability) are critical.

Technologies, tools and components used

• Languages & runtimes
  • Node.js (Express, Fastify, custom “Cpeak”)
  • C++ (Drogon framework) — final performance achieved in C++
  • Brief experiments with Python, Go, Java
• Load testing
  • autocannon (CLI + scripted JS wrapper)
  • Multi-machine orchestration via AWS SSM and small bash/node scripts
• Process management / clustering
  • PM2 for Node cluster-mode (spawn many Node instances per host)
  • Drogon (C++) uses internal threading
• Databases / caches
  • PostgreSQL (RDS/Aurora) for durable storage
  • Redis (standalone and Redis Cluster) for hot paths and ingest buffering
  • Migration tooling to move data Postgres → Redis for hot paths; background batch sync from Redis → Postgres
• JSON parsers
  • RapidJSON used in C++ (much faster than default Drogon parser)
• Monitoring
  • mpstat, top/activity monitor, system metrics, CloudWatch
  • Manual measurement of bytes transferred
• Cloud / infra
  • AWS EC2 (examples: local Mac Studio; C8i.32xlarge — 128 cores, 50 Gbit/s; C8GN.48xlarge “beast” — 192 cores, ~600 Gbit/s)
  • RDS instance types with tuned IOPS/provisioned throughput
  • S3 for results; AWS support/quotas (NLB capacity reservation; EC2 vCPU limits)

Key benchmarks / empirical results

Representative numbers from experiments and runs:

• Local Mac Studio single-route simple JSON
  • Express: ~14k–20k RPS
  • Fastify: ~66k–77k RPS
  • Cpeak (custom Node minimal framework): ~73k RPS
• Trivial endpoints on powerful tester + server
  • Millions of RPS possible when CPU and network are sufficient
• Patch route (heavier response ~30 KB)
  • Server becomes network-bound — moved multiple GB/s
  • Example peak ≈ 3M RPS when payloads made tiny and network was large enough
• PostgreSQL (writes)
  • ~30k–66k inserts/sec on a very large RDS instance
  • Increasing IO throughput improved RPS but cost rose sharply
• PostgreSQL (reads)
  • Bad query patterns (ORDER BY random(), SELECT COUNT(*) + random) scale terribly (O(n)) — seconds per response at millions of rows
  • Indexed/ID lookup design reached ~200k–400k RPS with careful queries and high-end DB instance
• Redis
  • Single Redis instance ≈ 100k RPS (read/write) typical bound
  • Redis Cluster (many masters + replicas) scaled to >1,000,000 RPS for the “code” route (UUID-based keys to avoid sequential contention); example: 30 master nodes, 15 replicas
• C++ Drogon + RapidJSON
  • Final optimized route reached ~1,000,000–1,200,000 RPS on beefy EC2 machines, handling huge aggregate throughput (example cited: 38 GB/s application-layer, ~300 Gbit/s)
• Final 30-minute large run
  • Architecture: 1 “beast” server receiving traffic from 60 small tester VMs (each running autocannon scripts)
  • Aggregate: ≈ 2 billion requests processed in 30 minutes, ≈ 60 TB transferred, ≈ 40 timeouts across billions of requests (very low error rate)

Architectural lessons, optimizations and recommended patterns

• Measure first — resource monitoring is essential (CPU core utilization vs aggregated CPU percent, network throughput, disk IOPS).
• Algorithmic complexity matters — avoid O(n) database patterns (random scans, COUNT(*)); prefer indexed lookups, precomputed metadata, and O(1)/O(log n) approaches.
• Reduce framework overhead — use a fast minimal framework for hot routes (Fastify or custom Cpeak over Express). For CPU-heavy hot paths consider native code (C++/Drogon, Rust).
• In-memory storage for hot writes/reads
  • Use Redis as an ingest buffer/queue for extremely hot routes, then batch-sync to durable DB asynchronously to reduce DB write load and cost.
  • Shard Redis (cluster) to scale past a single-instance Redis limit.
• Avoid sequential global counters/locks for hot inserts — use random UUIDs to remove single-writer contention (collision probability negligible at extreme scale).
• Networking is often the bottleneck for large payloads — provision adequate network bandwidth, use instances with high network performance, and reserve load balancer capacity where required.
• For maximal raw throughput prefer compiled languages + optimized parsers (C++ + RapidJSON used here).
• For production, use horizontal distribution across multiple servers/regions and geo-routing rather than relying on a single supercomputer-style box.
• Be cost-aware — RDS IOPS and instance/network capacity scale cost quickly; test carefully and avoid mistakes that multiply cost.

Engineering anti-patterns / gotchas uncovered

• Trusting default SQL code such as ORDER BY RANDOM or SELECT COUNT(*) at scale — massively underperforming.
• Relying on single-node Redis or single DB for very hot routes.
• Under-reserving cloud load-balancer capacity or hitting EC2 vCPU regional limits.
• Using slow JSON parsers or high-level frameworks without benchmarking for hot endpoints.
• Not accounting for TCP/OS limits (number of open connections) on the tester side — required spawning many smaller test VMs to saturate the server.

Product / project notes

• Cpeak: a zero-dependency Node.js mini-framework created by the author, optimized for performance and used in benchmarks.
• Demo product referenced: a URL-shortener project (“weer.pro”) intended to be stress-tested; open-source repos and code were shared to reproduce tests locally.
• Repos & scripts: multiple repositories provided (Node and C++ code, load-test scripts, Redis cluster scripts, migration scripts). The author recommends running locally before attempting cloud runs.

Costs and operational considerations

• Example instance costs (vary by region/time)
  • C8i.32xlarge ≈ $6/hr
  • C8GN.48xlarge ≈ $11/hr
• RDS instances with high IOPS and large storage can add thousands to tens of thousands $/month.
• Example totals
  • A high-throughput DB configuration might cost $7k–33k/month
  • Multi-VM tester setups add significant EC2 charges
  • Final experimental month cost ≈ $2,000 (could be reduced; mistakes would have increased cost)
• Energy & environmental note: moving terabytes/minute and sustaining many vCPUs consumes substantial power (comparable to driving an electric car many kilometers per hour in aggregate).

Practical how-to / guidance (recap)

• Benchmarks
  • autocannon options: c (connections), p (pipelining), d (duration), -w (workers/threads)
  • Concurrent in-flight requests = c * p
• CPU metrics
  • Understand per-core vs aggregated utilization (sum across cores can exceed 100% vs normalized 0–100%)
• Node clustering
  • PM2 to spawn processes equal to available cores
• Redis
  • Migrate Postgres data into Redis using batching; run local Redis cluster scripts; clustered Redis required to hit >100k RPS
• Testing tips
  • Use private VPC networking to avoid Internet throttles
  • Use multiple client machines to avoid bottlenecking the tester side

Final takeaways

• Achieving 1M RPS is feasible but requires:
  • Right language & parsers for CPU-bound handlers (e.g., C++ + RapidJSON)
  • In-memory buffering (Redis) and sharding for hot routes
  • Careful algorithm design (avoid O(n) DB ops)
  • Provisioning of sufficient network bandwidth and cloud resource reservations
  • Meticulous monitoring and cost-awareness
• The project is educational and reproducible: repos and scripts are shared so parts can be reproduced locally; careful testing is emphasized over blindly spinning up expensive cloud infra.

Main speaker / sources

• Speaker / presenter: the video author and builder of the demos (referred to in first person in the original content); provided repos and ran all experiments.
• Primary technologies / sources cited: AWS (EC2, RDS/Aurora, S3, CloudWatch, NLB, IAM), Node.js (Express, Fastify, Cpeak), autocannon, PM2, Redis (standalone & cluster), PostgreSQL, C++ (Drogon), RapidJSON.