Video summary

Jason Allum on Bedrock

Main summary

Key takeaways

Technology

Technology/Product Concepts Discussed (Bedrock Focus)

1) What Bedrock is trying to solve

  • Problem: Traditional key-value approaches (including Amnesia on the BEAM) struggle with dynamic cluster membership (adding/removing nodes) and other scalability/data-management limitations.
  • Goal: Provide an embedded distributed KV store with stronger correctness guarantees than the docs’ “beyond ACID” tagline.

Distributed consistency needs

Maintain a consistent “picture” across nodes while handling conflicts correctly:

  • Operations on independent keys should proceed in parallel.
  • Operations that touch the same keys should be ordered/serialized logically, so money/accounting-like operations don’t produce inconsistent results.

2) Bedrock mental model & API ergonomics

Bedrock keeps a familiar KV model:

  • put/get semantics
  • Range scans / walks over keys (more capable than a basic map)

Transaction model

Transactions resemble typical database transactions:

  1. A transaction reads keys
  2. Computes decisions
  3. Writes changes
  4. Commits or aborts

Elixir ergonomics

  • Transactions are typically expressed as a function/block
  • On conflict, the system can automatically rerun the transaction function

3) How conflicts are detected and resolved (FoundationDB-style)

Bedrock is described as “FoundationDB on the BEAM,” reusing the strategy (implementation details/language may differ).

Key mechanisms

  • Sequences / ordering: Each transaction gets a monotonically increasing sequence/timestamp to establish logical order.
  • Snapshot reads: Reads occur at a chosen read version, producing a consistent snapshot (“freeze in time” mental model).
  • Conflict detection via dependency tracking:
    • A commit is checked to see whether keys it read were modified in an intervening range/versions.
    • If a conflicting write occurred, the transaction is rejected and rerun.

No explicit row locking

  • Unlike Postgres-style locking, Bedrock uses optimistic concurrency:
    • conflict vs no-conflict rather than lock coordination.

4) System architecture & “single source of truth”

Instead of a single global master, Bedrock uses staged components:

  • Commit proxy: clients submit transactions here.
  • Resolvers: determine whether transactions conflict based on read/write sets and versions.
  • Logs: append/record committed transactions for ordering and durability (a “write-ahead log” style).
  • Storage/materializers: apply the transaction log to maintain readable state for versions.

Where “truth” comes from

  • Transactions are ordered via sequence/version
  • Once accepted, they’re recorded in the logs
  • Storage nodes answer reads for a given version
    • with a retention window (see the “5-second rule” below)

5) The “5-second rule” (version windowing)

To keep memory/state bounded, the system keeps only a recent time/version window (described as ~5 seconds).

  • Older history is squashed/flattened
  • Resolvers/storage do not need infinite version tracking

6) Reliability / recovery philosophy (OTP / supervision)

Rather than handling every distributed edge case with complex compensation logic:

  • Mark the current distributed runtime as invalid
  • Tear down and rebuild the distributed components
  • Restart/wire processes again

This is described as a distributed supervision tree approach (OTP philosophy extended across machines).

Leader election / singleton coordination

  • Uses consensus explicitly mentioned as Paxos
  • Raft is contrasted as “simpler/after”

7) Performance detail: zero-copy using binaries on the BEAM

Because processes don’t share memory on the BEAM, sending large data structures often causes copying.

Bedrock improves performance by:

  • Passing binary-encoded transaction packets between processes
  • Using the BEAM behavior where binaries > ~64 bytes can be passed efficiently (reference-counted / pointer passing), i.e., “zero copy”

Caution: Passing large maps/lists between processes can still incur copying costs; the binary strategy aims to avoid that.

8) Storage backends: logs → replicated storage / object store (S3/GCS)

Described pipeline:

  • Commit proxy batches transactions into blocks
  • Logs batch and persist to durable storage
  • Bedrock can use object storage such as S3/GCS for reliability/availability

Benefits

  • Reduces the need to run many dedicated storage servers for replication at the storage layer
  • Makes it easier to read materialized blocks cross-node

9) Layering and ecosystem potential (SQL/Ecto/Jobs/Workflows)

Bedrock is framed as a foundation layer (KV + distributed transactions), enabling higher-level systems:

  • SQL-like capability: FoundationDB research suggests SQL over a KV substrate.
  • Ecto integration idea:
    • An Ecto layer for FoundationDB exists/referenced
    • Lets Elixir apps compose queries while leveraging distributed transaction semantics.
  • Job scheduling layer:
    • A “job layer” on top of Bedrock (referencing Apple CloudKit/Quick-like work)
    • Distributed jobs that avoid double-running via transactional leases/conflict mechanisms.
  • Workflows/agents:
    • Jobs + transactions could support workflows/agents systems.

10) Use-cases emphasized

  • Distributed cache
    • Use Bedrock as a write-through cache
    • Postgres remains the system of record
    • Bedrock provides low-latency reads with consistent freshness.
  • Multi-region latency
    • Replicate/materialize data closer to where reads occur to reduce perceived lag.
  • Distributed correctness for shared-resource operations
    • Avoid races in account balance / debit / credit style flows.

11) Migration strategy / cautious adoption

Bedrock is described as bleeding edge.

Recommended adoption strategy:

  1. Treat it as a read-through / write-through cache alongside Postgres
  2. Gradually increase usage while working through issues

Key reassurance:

  • If the cache layer fails, fall back to the primary DB.

Speakers / Sources

  • Jacob Litzo — host / “Elixir mentor” (Elixir Mentor podcast)
  • Jason Allum — guest; creator/contributor related to Bedrock and beadwork

Additional referenced/mentioned sources (not main speakers):

  • FoundationDB
  • Apple CloudKit / Quick
  • Paxos
  • Raft
  • Amazon S3 / GCS
  • An Ecto layer for FoundationDB (referenced project)

Original video