Video summary

Making Materialized Views Actually Fast with DBSP (with Lalith Suresh)

Main summary

Key takeaways

Technology

Summary of Technological Concepts and Product/Approach Details

Core Problem: Incremental View Maintenance (IVM) Is Inefficient in Most DBs

A common database pattern is:

  • Materialized views/caches speed up reads
  • But when new data arrives, many systems recompute the entire query from scratch

For example, a query like:

  • SUM(amount) GROUP BY country

might read 10M rows to produce 12 summary rows. If a single row changes, rerunning the query from scratch is wasteful.

Some engines support incremental updates for “simple queries,” but the boundary of what counts as “simple” is:

  • Arbitrary
  • Often breaks as queries become more complex

DBSP/Faldera Breakthrough: Compile All SQL Into Incremental Form

The discussion centers on DBSP (Differential Dataflow / “DBSP” framing) and claims a key mathematical result:

  • Arbitrarily complex queries can be expressed such that updates process only the delta
  • Work becomes proportional to the size of the change, not the full dataset

The translation is described as:

  • Deterministic/syntactic
  • Not heuristic (i.e., no cost-based guesswork)

Unifying Batch and Streaming

DBSP/Faldera uses a single model for incoming changes:

  • Batch is just a large batch of changes
  • Streaming is a continuous flow of changes

Claimed outcome:

  • Exactly batch-equivalent semantics
  • Avoids “eventual consistency” behaviors common in some streaming systems

Key Data Model: Z-sets (Weighted Sets With Positive/Negative Weights)

Databases aren’t naturally “groups,” since insert/delete semantics don’t map cleanly to subtraction.

DBSP uses Z-sets:

  • Each row has an integer weight
  • Insert ⇒ weight +1
  • Delete ⇒ weight -1
  • Absent ⇒ weight 0

This makes “differences between states” tractable by computing changes via:

  • adding/subtracting weighted sets

System Architecture: “Changes Are First-Class Citizens”

Faldera is described as built from scratch (not bolted onto an existing DB) because it needs native understanding of changes.

In this architecture:

  • Tables, views, journals/updates, and outputs are represented in the Z-set/change-based model

How Incremental Computation Is Built: Core Operators

The video highlights four operators but emphasizes two main ones:

  1. lift (map-like): applies a function over elements
  2. delay (Z^-1): delays the stream by one time step, enabling feedback circuits used for:
    • integration
    • differentiation

With lift + delay, the system can build:

  • stream differentiation (turn stream/state into “changes of”)
  • stream integration (reconstruct state from changes)

Higher-level claim:

  • SQL can be compiled into circuits composed from these operators

State and Indexing Tradeoffs

Some operations can be incrementally updated without storing complex history (linear/non-stateful), such as:

  • filters
  • projections
  • linear aggregates like SUM/COUNT

Other operations require maintained state via delay, typically involving:

  • joins
  • distinct
  • group by
  • certain aggregations

Framed as a trade:

  • More storage/index state
  • In exchange for much lower recomputation cost

Join Behavior: “Delta Join” Reconstruction

The explanation describes join updates as:

  • compute delta effects from the left changes and right changes
  • combine those effects with the prior join result

The join circuit is said to automatically recover the delta join algorithm (a known database optimization concept).

Materialized Views: “Reconstitution” vs Delta Propagation

The system distinguishes between:

  • views that propagate deltas
  • materialized views that may require reconstituting full results
  • local views (not externally observable) that enable more aggressive optimizations

If there is no primary key / not declared as materialized, the example suggests the system can:

  • avoid maintaining state and
  • simply push deltas through

Operational Model: Synchronous Compute Rounds

Inputs supply batches of changes each “next round.”

For each round:

  • the engine evaluates
  • produces the set of outputs that changed
  • described as atomic per round

Connectors interface with external systems, while internally everything is expressed as:

  • applying/appending changes

Backfills (“Huge Steps”)

For initial population or large historical loads:

  • backfills can be treated as one gigantic transaction
  • producing one large delta downstream

This avoids expensive repeated incremental steps during backfill.

UDF Support Constraint: Determinism Only

Pipelines require deterministic functions (pure computation). Non-determinism is disallowed, such as:

  • randomness
  • external API calls

UDFs/UDAFs are compiled into Rust binaries using the DBSP crate.

now / Temporal Semantics

  • now is modeled as a clock/time stream input connector
  • Warning: using now in some contexts (e.g., projections) can become an anti-pattern since it makes outputs time-varying per record

During checkpoint recovery:

  • now reflects the stream value at resume
  • outputs remain correct under the model

Claimed Performance / Review-like Takeaways

  • Incremental work grows with change size, not total dataset size.
  • Reported benchmark example: 48 hours → 2 seconds at 50× scale (team-reported).
  • Transition target: from multi-hour/nightly recomputation to milliseconds/seconds while staying fresh.

Guides / Tutorials Mentioned

  • Try it: try.faldera.com
  • Open source: github.com/felder
  • Code/library: a Rust DBSP crate underlying compilation to incremental circuits
  • Talks suggested:
    • Confluent talk (2024): “Streaming Queries without Compromise”
    • CMU database group talk: deeper dive into the paper (about an hour)

Main Speakers / Sources

  • Chris Jenkins (host, Developer Voices)
  • Lilith Suresh (CEO of Felder/Faldera, discussing DBSP and Faldera’s approach)

Referenced source materials:

  • the award-winning DBSP paper
  • related work such as delta join and differential dataflow
  • mentions of Felder, DBSP, and Calixte/Apache Calcite as a front-end

Original video