Lefteris Kokoris-Kogias - Coding Techniques in Walrus for Massive Storage

Overview

• Context: Interview with Lefteris Kokoris-Kogias (researcher at Mist Labs, contributor to Sui) about engineering and research work that connects blockchains and decentralized storage: Walrus (decentralized storage), Seal (privacy / key-management layer), and related research (consensus, sparse nodes).
• Core message: Build modular, open-source infrastructure pieces (focus on doing one thing very well) and compose them. Use Sui as a performant control plane for coordination while Walrus serves as the data plane for efficient decentralized blob storage.

Mist Labs and Sui

• Mist Labs: an infrastructure company focused on practical, open research and production systems for decentralization.
• Sui: a Move-based layer-1 blockchain used by Mist as the control plane for Walrus. Smart contracts on Sui handle storage marketplace logic, committees, reconfiguration, and related coordination. Sui was chosen for performance; Walrus itself is an independent storage system and can be redeployed to other chains.

Walrus — decentralized storage

Role and goals

Walrus is the data plane optimized for decentralized storage of large blobs (websites, NFTs, videos). Emphasis areas:

• Efficiency and practical usability (APIs and S3-like flows).
• Reconfiguration and decentralized repair.
• Low-cost per-GB storage and open access (ability to move data if desired).

Architecture

• Clear separation between control plane (smart contracts on Sui) and data plane (Walrus storage nodes).
• On-chain decisions include buying/trading storage, committee membership, and reconfiguration.
• Walrus focuses on availability and consistency instead of heavy cryptographic content proofs.

Two-dimensional erasure coding

• Uses a Reed–Solomon-like erasure coding arranged in a 2D grid (rows × columns) rather than a single linear encoding.
• Each storage node holds one row and one column; intersection cells add redundancy.
• Benefits:
  • Self-healing and efficient partial recovery: if a node loses some data, it can request small pieces from peers instead of re-downloading whole files.
  • Enables efficient decentralized re-repair and reconfiguration in a quorum-based system.
• Trade-offs:
  • Higher network/storage overhead: reported roughly 4.5–5× replication vs ~3× for single-dimension Reed–Solomon.
  • Increased overhead is accepted in exchange for much easier decentralized repairs and reconfiguration.

Commitments and verification

• Encoder commits to rows/columns using Merkle-tree-style metadata; current verification is per-row.
• Finer-grained proofs (e.g., vector commitments) are possible but increase metadata size.
• If an encoder persistently produces garbage, the system flags data as unavailable and a block/operation can be skipped.

Reconfiguration and recovery

• Epoch-based committee rotation discourages arbitrary data movement.
• When handover is needed, new nodes request specific cells from peers to reconstruct only missing pieces (partial recovery).
• Overlapping epochs and rules limit adaptively malicious behavior; rewards or funds can be used to pay for recovery if nodes fail to hand over properly.

Comparison to prior work

• Related ideas (e.g., 2D concepts) have appeared in projects like LazyLedger (sampling for availability).
• Walrus prioritizes decentralized repair and practical reconfiguration in a quorum-based environment rather than fully open sampling-driven availability systems.

Use cases and economics

• Target uses: durable NFT asset storage, decentralized websites (Walite), CDN-like use cases, general blob storage replacing vendor lock-in S3-style workflows.
• Emphasizes low-cost storage and developer-friendly workflows (S3-like APIs planned).

Seal — privacy and key management

• Purpose: provide a usable privacy / key-management layer without forcing every user to manage symmetric keys locally.
• Design: multi-party computation (MPC) servers run a protocol to perform key management for clients, enabling client-side encryption workflows with server-assisted key custody/operations.
• Properties and trade-offs:
  • Generic protocol applicable beyond Walrus; being deployed to mainnet.
  • Improves privacy UX, but requires trusting MPC servers (some additional trust vs fully local key management).
  • Can support multiple purposes, including data encryption and potential custody operations (with trust caveats).

Research and related work

• Lefteris’s background: academic distributed-systems research with work connecting BFT and proof-of-work ideas; transitioned to building practical systems.
• Papers and open research:
  • Walrus: paper published in a reputable venue; implementation is open source and had a mainnet deploy.
  • Seal: deployed or deploying on mainnet.
  • Sparse nodes: draft/preprint (authors include Deepak, Carl, Alberto with collaboration from Julius Cafino) enabling verification of a substate of an application cheaply (lightweight “sparse” verification instead of running a full node). Sparse nodes aim to make state portability and replayability feasible, helping avoid lock-in.
• Design philosophy: modularity (Unix-like) — build focused, composable components rather than end-to-end vertical stacks; open science and open source to foster ecosystem collaboration.

Security and adversary model

• Walrus uses a quorum / trusted-validators model rather than an open permissionless sampling model.
• Availability and integrity reasoning rely on quorum-based guarantees and reputation/committee mechanisms.
• The 2D coding plus epoch overlap approach constrains adaptive adversaries; recovery protocols limit what an adversary can force (communicating only missing parts rather than full-object reconstruction).

Operational and ecosystem notes

• Walrus is open-source; grants and developer tooling are available.
• Integrations and deployments to other chains are possible, though the mainnet instance currently uses Sui.
• Mist emphasizes engineering: research-informed production systems aimed at widely usable storage for developers and institutions.

Resources and pointers

• Walrus paper (published in a reputable venue) — link referenced in show notes.
• Sparse nodes preprint/submission — authors listed and preprint submitted.
• Walrus mainnet: deployed (March).
• Seal: deployed / deploying on or near mainnet.
• Open-source code, grants, and APIs (S3-like interfaces expected).

Main speakers and contributors

• Lefteris Kokoris-Kogias — guest; researcher at Mist Labs; contributor to Sui, Walrus, and Seal.
• Host: Joni Syndro (Common Prefix / Honest Majority podcast).
• Organizations: Mist Labs, Sui Foundation, Common Prefix.
• Coauthors / contributors: Julius (Julius Cafino), Deepak, Carl, Alberto, and George Danezis (Mist’s chief scientist).

Notes on transcription

Some names/words in auto-generated subtitles may be slightly mistranscribed (for example, “rich Solomon” = Reed–Solomon). Minor name or spelling errors (e.g., “Misten/Mistton”) may appear in raw subtitles.