Distributed SQLite with Litestream and LiteFS

Summary of technological concepts & product features

Context / motivation (SQLite + embedded databases)

• The speaker works on systems at Fly.io and previously built Litestream, aimed at making SQLite suitable for reliable backups (especially “disaster recovery”).
• They emphasize that simple embedded SQLite deployments can go a long way.
• Many apps only need modest uptime—so the focus is minimizing data loss, not implementing full enterprise HA complexity.

---

Litestream (distributed-ish durability for SQLite single-node apps)

Core idea

• Continuously stream incremental SQLite changes to remote object storage (e.g., S3-compatible).
• Reduce potential data loss compared to hourly/daily backups.
• Runs as a separate long-running process, so the application doesn’t need modification to “know” about replication/backup.

Why it matters vs regular backups

• SQLite journaling modes:
  • Rollback mode: traditional approach; less concurrency. Recovery uses a journal file.
  • WAL mode (Write-Ahead Logging): typically preferred for concurrency (many readers, one writer).
• In WAL:
  • Replication must track changes since the last incremental backup.
  • The replicated stream must align with checkpointing.
• Key risk described:
  • If a disaster happens around a checkpoint, WAL can be cleared.
  • The last small transaction window may become undetectable, causing larger-than-expected data loss.
  • In some cases, recovery can be incorrect if you can’t reconstruct what happened.

Key technical “hack” / architecture

Litestream leverages WAL behavior without modifying SQLite internals by:

• Keeping a long-running read transaction open to prevent SQLite from discarding WAL content Litestream still needs.
• When checkpoint/backup is “ready,” Litestream coordinates to capture WAL data safely, then resumes the read transaction.

Checkpoint readiness + WAL handling

• Litestream effectively mirrors SQLite’s WAL threshold behavior (example: around 4MB), when SQLite attempts to checkpoint.
• It triggers SQLite’s checkpoint logic rather than re-implementing checkpoint internals, using knowledge of the documented WAL format.

Shadow WAL for resiliency (object storage downtime)

• Litestream uses a “shadow WAL” directory next to the DB.
• If the replication target storage (S3/S3-compatible) is temporarily unreachable:
  • Litestream can stash incremental WAL/checkpoint-related data.
  • This allows checkpointing and application writes to continue without blocking or corrupting the replication pipeline.

Storage layout in S3

• Two main subdirectory concepts:
  • Snapshots (default described as daily)
  • WAL / incrementals (incrementing files replayed in order)
• Recovery can reconstruct database state at any time between:
  • a snapshot, and
  • subsequent WAL increment files.

Restore / recovery workflow

• CLI restore command:
  • litestream restore ... (restore snapshot + replay incremental WAL files until the desired time)
• Restore time depends on the number of incremental files.
• Recommendation:
  • Consider more frequent snapshots if there are many changes.

Compression

• Incrementals are described as compressed (including LZ4 compression).

---

LiteFS (distributed replication, failover, and multi-region replicas)

Core goal

• Provide distributed systems replication for SQLite, suitable for:
  • Failover during deployments (reduce downtime)
  • Read replicas at low latency across regions

Why it’s different from Litestream

• Litestream: framed as “disaster recovery only” (single-node streaming backups to object storage).
• LiteFS: distributed replication across multiple nodes—more complex because correctness under concurrency and failure matters.

How LiteFS “sneaks in” without modifying SQLite

• LiteFS acts as a pass-through user-space file system via FUSE.
• It works by:
  • Mounting a filesystem directory that the application writes to.
  • SQLite believes it’s writing to a normal file, but writes go through LiteFS.
  • LiteFS forwards writes to the real filesystem and also captures changes.

Transaction packaging for replication

• LiteFS uses SQLite file locks to detect transaction boundaries (start/end transaction).
• It packages each transaction’s changes into an “LTX transaction file”.
• Replication:
  • A primary node receives and stores changes.
  • Replicas connect to the primary and stream LTX transaction files continuously to replay in order.

Integrity / corruption detection

• LiteFS uses rolling checksums (described with an XOR-style approach) for pages involved in each transaction.
• Purpose:
  • detect corruption
  • validate that replicas match

Leader / primaries and failover options

• Multiple-primary behavior can be supported:
  • either a fixed primary
  • or leader election (example tooling mentioned: Consul)
• Failover correctness challenges discussed:
  • Leader election can acknowledge a write before it reaches the candidate.
  • This can require checksum-based resync/re-snapshot to ensure correctness.

Read consistency / avoiding out-of-order effects

• Concern:
  • writes sent to the primary may not reach a replica before reads occur,
  • causing replicas to serve stale data.
• Approach:
  • An optional HTTP proxy can enforce that reads wait until the replica has applied up to the relevant write transaction.
• This is described as “sandwiching” the app between:
  • proxy + filesystem,
  • giving application-consistent reads without per-read transaction ID checks.

Deployment / usage notes

• LiteFS is available via Fly.io documentation and packaged as a Docker image/bundle.
• Typical approach described:
  • copy the LiteFS binary into the Docker image.
• Integration idea:
  • point HTTP-access apps (e.g., WordPress) at LiteFS so routing handles reads/writes appropriately.

---

Future direction and positioning

• Liteestream

  • Intentionally scoped: streaming backups to remote storage for disaster recovery.
  • Not aiming to become a full distributed replication system.

• LiteFS

  • Focus on performance, reliability, and incremental features rather than “feature creep.”
  • Mentioned possible additions:
    • an HTTP query API for querying LiteFS over HTTP
    • “LiteFS Cloud”: a paid service to store backups remotely and support point-in-time recovery

---

Main speakers / sources (as indicated in subtitles)

• Ben Johnson (interviewer / speaker shown early)
• Ben (Fly.io engineer/creator of Litestream and LiteFS; name not explicitly repeated in subtitles)
• Sources/tools mentioned by name:
  • SQLite, S3, GCS, Azure, WAL, FUSE
  • Prometheus/Grafana
  • Consul
  • Fly.io
  • LZ4 compression
  • LTX transaction files