Video summary

How Instagram Scaled Postgres to 2 Billion Users

Main summary

Key takeaways

Technology

Technological concepts & scaling takeaways (Postgres at Instagram / “planet scale”)

Why Postgres stayed the choice

Systems like Instagram, Reddit, Notion, Heroku, Strava, Discord reportedly stuck with Postgres even though alternatives (e.g., NoSQL) existed. The core claim is that the bottlenecks weren’t fundamentally “Postgres can’t scale”—they were operational or architectural issues that could be addressed while remaining relational.

Early Instagram architecture (2010–2012)

Instagram began with a simple setup:

  • EC2 for the app
  • S3 for photos
  • A single Postgres database holding:
    • accounts
    • metadata
    • comments/likes
    • follow graph

Even by ~27M users (around the Facebook acquisition), they were still using a single Postgres DB (about 2 TB) with limited engineering headcount—supporting the idea that Postgres can handle large workloads with the right engineering.

Key bottlenecks and concrete fixes

1) Connection scaling wall (first “invisible” bottleneck)

Problem

  • Django app servers created many database connections.
  • Each connection was about ~1.3 MB, and the example given was:
    • 50 app servers × 30 connections each ≈ 1500 connections
  • That consumed about ~2 GB RAM before Postgres did meaningful work.

Fix

  • Use connection pooling, specifically PG bouncer.
  • PG bouncer works as a lightweight proxy between apps and Postgres, multiplexing many incoming app connections onto fewer real Postgres connections.

Review-style guidance

  • If running Postgres at meaningful scale, add PG bouncer early, described as the highest leverage change.

2) Vertical scaling wall (single-machine limits)

Problem

  • After connection pooling, Instagram hit the next limit: a ~2 TB database on the largest EC2, maxing RAM and disk I/O and leaving no room to grow on one machine.

Common advice discussed

  • Switch to NoSQL because relational systems “don’t scale horizontally.”

Instagram’s decision

  • They evaluated NoSQL options but concluded their issues were workload/scale problems that any database would face.
  • Switching to Cassandra would require “burning the boats,” including complexity around partitioning, without solving the fundamental bottleneck category.

Horizontal scaling pattern: sharding Postgres (but the “right” way)

Shard-by choice: user ID tradeoff

Why shard by user ID

  • User-centric queries are fast when each user’s data lives on one shard.

The catch

  • Social feed assembly is cross-user (e.g., “photos from people I follow”).
  • If followed users span many shards, assembling the feed requires:
    • fanout
    • merging/sorting
  • This causes higher latency and more failure modes (e.g., one slow shard delays the whole feed).

Key tradeoff

  • Single-user queries are fast.
  • Cross-user queries become expensive/complex.

Mitigation

  • They accept this and reduce impact with application-layer techniques like:
    • caching
    • smart fanout
    • precomputed feeds

The “clever architecture” part: separate logical shards from physical placement

Many sharding approaches couple shard count directly to hardware (add servers → reshuffle).

Instagram’s approach:

  • Create thousands of logical shards (Postgres terms: schemas).
  • Each logical shard contains the same table set (users, photos, likes, etc.) but holds different data slices.
  • Initially, all logical shards may live on one physical machine.
  • Mapping is configurable and changeable: logical shard → physical node can be updated.

Scaling benefit

  • When a physical node runs out of space:
    • move logical shards to a new machine using streaming replication
    • update a mapping table
  • No rewrite of user data, and the app doesn’t need code changes because schemas and access patterns remain consistent.

Main pattern summary

  • Separate data partitioning (logical) from data location (physical); growth becomes a mapping/config change rather than a full reshard rewrite.

IDs wall in sharded Postgres, and Instagram’s solution

Why auto-increment IDs break immediately

  • Multiple shards might generate IDs from the same sequence (e.g., 1, 2, 3…).
  • Without coordination, ID collisions occur, breaking application assumptions.

ID generation options evaluated

  • UUIDs
    • collision-free in practice
    • but larger storage footprint
    • random ordering makes “latest items” queries harder/less efficient
  • Ticket server (Flickr-style)
    • a dedicated service to mint unique IDs
    • rejected due to a single point of failure
  • Snowflake-style
    • time-ordered distributed IDs
    • usually requires an external service
    • Instagram’s twist: implement Snowflake-like IDs inside Postgres

Instagram “Snowflake-style in Postgres” (as described)

  • 64-bit IDs composed of:
    • 41 bits timestamp (milliseconds since a custom epoch)
    • 13 bits shard ID (up to ~8,192 logical shards)
    • 10 bits sequence (per shard per millisecond; up to ~1024 IDs/ms)
  • Implemented as a Postgres function, so:
    • no external ID service
    • no single point of failure
    • IDs are roughly time-sortable, enabling “latest photos” by ordering on the ID

“Post features” highlighted (product/DB feature utilization)

  1. Partial indexes

    • Index only rows matching a predicate (e.g., only last-30-days photos).
    • Benefit: index size shrinks, improving performance as old data ages out.

  2. Functional indexes

    • Index computed values rather than entire large strings.
    • Example: for long random tokens, index only the first 8 characters to reduce index size while still supporting efficient lookups via unique prefixes.

  3. Logical replication

    • Stream inserts/updates/deletes to downstream systems.
    • Intended pattern:
      • Postgres publishes changes (e.g., photos table)
      • search index subscribers update automatically
      • cache invalidation follows streamed changes
      • analytics warehouse maintains near-real-time copies
    • Emphasized as a more “built-in” alternative to manually wiring event pipelines (Kafka/dual writes).

Meta-evolution note (historical vs modern)

  • The scaling story is framed historically (roughly 2010–2015).
  • Later, as Meta/Instagram integration evolved, the social graph is served by “Tao”, a Meta-built distributed graph system—i.e., not Postgres as the primary substrate.

Main takeaways (explicit guidance)

  • Don’t shard until you absolutely have to (Instagram waited until very large user counts).
  • When sharding:
    • separate logical shards from physical placement
    • use many logical shards
    • treat physical mapping as configurable
  • Use Snowflake-style IDs early, even before sharding, to avoid painful migrations later.
  • Connection pooling (PG bouncer) is the biggest immediate win for high-scale Postgres.
  • “Boring technology wins”: adopt new infrastructure only when you face genuinely new categories of problems; Postgres works if you understand and solve the real bottlenecks.

Main speakers/sources

  • Primary speaker: the video’s host/author (also promotes their course/curriculum), explaining the technical approach throughout.
  • Mentioned systems/companies as sources/examples: Instagram, Meta (Tao), Reddit, Notion, Heroku, Strava, Discord, Flickr, Twitter (Snowflake), plus Postgres features (no specific external author cited for those).

Original video