How Uber serves 40 million reads per second

Summary of technological concepts & product/engineering features

Core scaling problem at Uber

• At large scale, the main issue is not hacking or generic cloud outages.
• The central challenge is handling self-generated traffic overload.
• A common failure pattern is the “thundering herd”:
  1. Slow partitions cause client timeouts
  2. Clients retry
  3. Retries multiply load
  4. A localized spike becomes a platform-wide incident

Uber’s distributed database architecture

Uber serves tens of millions of requests per second using thousands of microservices for 170M+ monthly active users.

Two in-house distributed databases

• Schemaless: an append-only design originally built for trip-related data
• Docstore: later evolved into a general-purpose transactional DB used across Uber verticals

Common foundation

Both databases are built on top of MySQL storage and include:

• a stateless query engine for:
  • routing/sharding
  • parsing
  • authorization
• a stateful storage engine where:
  • data is sharded into partitions
  • each partition uses 1 leader + 2 followers
  • coordination is done via Raft
• storage backed by MySQL

Key takeaway: despite “exotic” names, the foundation is MySQL + consensus, combined in sophisticated ways.

Database evolution: Schemaless → Docstore; Postgres → MySQL

Uber initially evaluated Cassandra/MongoDB, but needed capabilities including:

• writes during failure
• notifications to downstream systems on data changes
• secondary indexes

Schemaless design intent

• Built to scale trip data horizontally away from a single Postgres instance.

Moving from Postgres to MySQL (2016)

Uber moved largely due to write amplification in Postgres:

• Postgres updates create new row versions
• secondary indexes reference physical tuple locations
• this increases:
  • WAL writes
  • disk I/O
  • later maintenance (e.g., vacuum)
• replication compounded the effect:
  • amplified storage changes were carried over physical replication
  • replicas lagged across data centers

Why MySQL / InnoDB fit better

• InnoDB better matches Uber’s update/index patterns because:
  • secondary indexes reference the primary key, not physical tuple locations

---

Overload protection / load shedding journey

1) Quota-based rate limiting (initial approach — failed)

Implemented at the query engine layer:

• assigns a “capacity cost” per request (e.g., bytes processed)
• enforces quotas per user
• returns HTTP 429 when exceeded

Why it failed:

• Centralization problem: stateless routing nodes required Redis for quota metering
  • adds an extra DB/cache hop per request
  • introduces a single point of failure
• Bad metering model: MySQL query billing couldn’t distinguish query costs
  • table scans vs single-row access could be billed similarly
• Static quotas: teams continuously requested higher quotas
  • quotas became effectively non-enforcing (“mandatory quota fully optional”)

2) Controlled delay + concurrency-based overload management

Overload management moved closer to storage nodes to use better context.

Key shift:

• from measuring QPS to measuring concurrency (in-flight operations)
• aligned with Little’s Law: concurrency = throughput × latency

Codel (networking-inspired)

• based on controlled delay (how long requests wait), not queue length
• separate queues for:
  • reads
  • writes
  • background work
• normally FIFO, but under pressure:
  • switches to LIFO
  • rationale: older requests likely timed out and were retried; newer requests still have an active client

Scorecard (tenant admission control)

• rule-based admission control that caps per-tenant concurrency
• designed to prevent “noisy neighbors”

Node-local regulators (detecting hidden overload modes)

• detect issues like:
  • IO saturation
  • hot partition keys
  • low memory
  • goroutine-count pressure (throttle when too many goroutines start)

3) Cinnamon (priority-aware overload shedder using PID control)

Why Cinnamon replaced Codel

• Codel was blind to priority
  • important and non-important work were shed with equal chance during overload

Cinnamon design

• each request carries a priority tier:
  • tier 0 = critical infrastructure
  • down to tier 5 = low-value workloads
• shedding occurs bottom up during overload
  • critical writes remain alive while low-priority work is dropped

PID controller-based shedding

• Cinnamon chooses shedding amount via a PID controller:
  • Proportional / Integral / Derivative
• uses feedback history/trends to smoothly converge to a stable rejection rate
• this avoids token-bucket “hammer” behavior that can cause:
  • retry storms
  • oscillation

Unified overload engine with “bring your own signal”

• Cinnamon becomes a general overload engine with pluggable signals
• overload decisions can be non-local:
  • e.g., leaders may look healthy but shed because Raft followers are falling behind
• commit/replication lag signals feed into the same control loop as memory/concurrency signals

Reported results

• 80% more throughput under overload
• coroutine count during incidents down 93%
• heap spike reduced from ~5–6 GB to about 1 GB

---

Key “failure mode” analogy / trivia (analysis framing)

• The video references AT&T’s 1990 long-distance network collapse as a cautionary example.
• The system became unstable due to recovery traffic amplification:
  • crashing switches broadcast overload/out-of-service messages faster than neighbors could handle
• This maps directly to Uber’s focus on avoiding retry/recovery traffic storms.

---

Main speakers / sources (as indicated by the subtitles)

• Video host / narrator:
  • introduces “The Blueprint”
  • explains Uber’s architecture
• Uber engineering teams / published engineering posts (primary source material), especially:
  • posts on Schemaless / Docstore
  • posts about moving from Postgres to MySQL
  • design details of Cinnamon (noted as originally built by Uber’s delivery team)