Принципы проектирования высоконагруженных приложений

Main ideas and lessons (clear outline)

1. High-load systems often have a “bias” toward either reads or writes

   • Applications can be read-intensive or write-intensive, meaning one dominates resource usage.
   • In the speaker’s experience (distributed tracing at Yandex), the system was heavily write-intensive:
     • ~30–40k RPS on the write side
     • total ~10–11 GB/sec of traffic
     • stored/processed into a large ClickHouse database (described as petabytes-scale)
   • Key lesson: imbalances that are invisible in small systems become critical as load grows.
   • Another lesson: you must consider expected load; otherwise you can design in the wrong direction.

2. Simple solutions become expensive or ineffective under heavy load

   • Example A: Preventing data loss via a message broker
     • A straightforward approach would be: services → Kafka → batch write to ClickHouse.
     • But the estimated Kafka cluster size was too large (the speaker cites needing on the order of ~1000 “dashes” of 2 cores, plus RAM/disk/network costs).
     • Result: the “obvious” solution was rejected due to cost; they accepted tradeoffs instead.
   • Example B: “Write to both hot and cold storage” seems simple but doesn’t scale
     • Naively writing to two storages immediately would duplicate massive data volumes.
     • The scalable alternative was:
       • store initially for ~24 hours
       • then move data to cold storage in a separate process (avoids duplication across petabyte-scale volumes).

3. “Clean architecture” / API-only interaction can still be suboptimal at scale

   • Principle (attributed to Jeff Bezos): services should communicate only via APIs, with no direct database access.
   • The speaker agrees for most cases, but argues there are exceptions under heavy load:
     • If one service must serve large outgoing traffic, routing through an extra service adds:
       • additional CPU/processing load
       • extra network hop bandwidth cost
   • Under resource pressure, it may be more effective (not “conceptually pure”) to allow direct database access in rare/high-stakes scenarios to reduce network hops.

4. Do not spread a large workload across the whole system—concentrate it

   • Example: Tinder/VK Dating-like flow (simplified)
     • Profile service sends profile batches (~5,000 RPS, ~20 per batch)
     • swipe/match service processes swipes/matches (~100,000 RPS total)
     • profile service needs “likes” to avoid showing already-liked profiles
   • Naive approach: write likes handled through multiple services → spreads 100k RPS across subsystems.
   • Better concentration concept:
     • collect likes into an intermediate storage (Redis suggested as one option)
     • later aggregate into “lists” that profile generation can query efficiently
   • Key lesson: concentrate heavy write/read workloads in a dedicated place, rather than scattering them across many services.
   • If you can’t concentrate, reduce real-time load (e.g., batching/aggregation, delaying non-critical work).

5. Fault tolerance must consider data center–level reality, not just replication

   • Speaker describes a multi–data-center setup with master replication and asynchronous replication.
   • Problem: if a data center goes down, it may still accumulate telemetry/log/span data.
   • Typical debugging wants the latest data, but when a center is down, you must reason about:
     • what telemetry is delayed
     • how to use it to diagnose the cause
   • Lesson: data center fault tolerance is more than setting up cross-data-center replication; it must support observability and incident debugging needs.

6. Optimization is effectively never-ending

   • Recurring bottleneck pattern:
     • solve one bottleneck → another appears elsewhere
   • Lesson: expect a continuous cycle of bottleneck discovery and resolution.

7. Performance ≠ scalability

   • Performance: how quickly the system processes requests right now.
   • Scalability: how well it handles increasing load over time.
   • A system can be:
     • fast on one server but fail when load increases (not scalable)
     • less fast initially but scale better with growth

---

Methodology / principles expressed as actionable guidance

A) Design around read/write bias

• Determine whether the system is:
  • Read-intensive or Write-intensive
• If the bias is insignificant, you can sometimes ignore it.
• If the bias is significant toward reads:
  • minimize actions during reading
  • shift work to writes/prepare for fast reads:
    • aggregate during writing
    • cache data for faster reads later
• If the bias is significant toward writes:
  • minimize actions during recording/writing
  • accept extra work during reads:
    • store data in a “not-so-smart” way to speed ingestion
    • do additional processing/aggregation later when reading

B) Beware “simple” reliability or storage patterns at scale

• Don’t assume the naive approach works under petabyte-scale/high RPS.
• Validate by estimating resource needs (compute/storage/network).
• If a reliability add-on is too costly (e.g., Kafka):
  • consider alternative tradeoffs rather than direct implementation.
• For hot/cold tiering:
  • avoid duplicating petabytes by writing to both immediately
  • prefer staged migration (e.g., keep hot for a day, then move via batch process)

C) Be pragmatic about architecture principles under heavy load

• Start from clean principles (API boundaries), but:
• Under extreme load/resource constraints, be willing to break “purity” if it reduces:
  • extra processing in intermediate services
  • extra network hops / bandwidth costs
• Make such decisions as exceptions, not defaults.

D) Concentrate workload; reduce or batch when possible

• Prefer architecture where:
  • heavy incoming workload is collected in a dedicated component
  • subsequent consumers read aggregated/prepared results
• If concentration isn’t possible:
  • reduce real-time requirements via batching and asynchronous aggregation
  • only do “must be real-time” actions synchronously

E) Plan fault tolerance with observability in mind

• Treat data center failure as an incident + debugging workflow problem:
  • replication alone may not solve troubleshooting needs
  • ensure telemetry/log collection supports debugging with near-latest data

F) Expect iterative optimization

• After addressing one bottleneck, plan for new bottlenecks to emerge.
• Run continuous measurement → hypothesis → test → optimize loop.

G) Measure and design for scalability separately from performance

• Evaluate:
  • latency/throughput under current conditions (performance)
  • behavior under increased load (scalability)
• Don’t assume fast implies scalable.

---

Speakers / sources featured

• Unnamed speaker (the presenter; no name given in the subtitles)
• Jeff Bezos (referenced for the “services interact via APIs, no direct database access” principle)
• YouTube Developers / an article (referenced) (used to illustrate “bottlenecks keep moving” during optimization; not a specific named person)