Scaling Kafka at Blinkit & Zomato - Confluent DSWT 2026

Summary

Pradeep Khileri, a site reliability engineer at Blinkit, describes two ways Eternal’s data platform—serving Blinkit, Zomato, District, and other businesses—uses Kafka-compatible streaming technology to handle scale and reduce operational work.

1. Moving from self-managed Kafka to WarpStream

• The observability platform handles a reported peak of about 25 GB/s of uncompressed logs, or roughly 1 billion records per minute. Typical records are around 1–1.5 KB.
• Previously, the team operated its own Kafka cluster, including 24 brokers in an earlier setup. Scaling for unpredictable peaks meant paying for spare capacity and managing tasks such as adding brokers, rebalancing partitions, and handling replication.
• The team adopted WarpStream’s separation of compute from storage:
  • Kafka-compatible compute agents handle producer and consumer requests.
  • Data is stored in S3 rather than on broker-local disks.
  • A separate control and metadata layer handles coordination and offsets.
• The talk describes separate compute for business units and a data-platform team that reads the data for analytics. The speaker says tenant data remains segregated, while shared platform infrastructure makes it easier to access and process logs without repeatedly copying them between clusters.
• The reported result was an estimated 60% cost reduction, alongside less cluster-management work. The speaker also notes that WarpStream’s agent binary is not open source.

2. Using compacted topics for batch processing

• Blinkit has many changing records, such as inventory, ledger, and product-catalog updates. Some downstream jobs, however, need only the latest state of each record.
• The team uses Kafka log compaction to make topics behave somewhat like tables containing the latest value for each key. The speaker describes configuring infinite retention and a compact cleanup policy, with compaction-lag settings governing when records become eligible.
• Batch and indexing jobs can load the compacted state, apply business rules, and rebuild indexes rather than repeatedly gathering updates from multiple services.
• WarpStream’s workload separation is used to isolate normal streaming traffic from batch reads and compaction. A separate compaction agent reads data from S3 rather than competing with regular producers and consumers for the same broker CPU.
• The reported indexing time fell from about five hours to 30 minutes. Keeping the data available in S3 also makes it easier to test or rerun indexing and rebuild systems such as Elasticsearch or Redis without first mirroring production data into another cluster.

Operational practices and Q&A

• Recovery and replay: Consumers can reset offsets to reread data. For large backlogs, the speaker describes copying or fanning data out from a smaller number of partitions to more partitions, then scaling consumers in parallel. Recovery time depends on service-level agreements and backlog size.
• Idempotency: Replaying records can repeat side effects, such as issuing a refund twice. Consumers should check whether an operation has already been processed, using business-state checks or unique record IDs, potentially backed by a cache.
• Storage and offset consistency: In response to a question about data reaching S3 before offsets update, the speaker says WarpStream’s control plane manages consensus and offsets using DynamoDB, and describes its reliability as part of the managed service. He does not explain the implementation in detail.
• Ordered updates: For workflows where several services update the same order, the speaker recommends converging events into an ordered Kafka stream before updating state. This can avoid concurrent database writes and related deadlocks.
• Mirroring and offsets: The speaker explains that MirrorMaker’s source and checkpoint connectors track offset translation between clusters. He also recommends idempotent consumers when migrating or replaying data.
• Consumer throughput: Polling and batch size depend on the workload. The team uses an internal Kafka Go library and adjusts settings to match each service’s SLA.
• Monitoring: The team monitors agent-level metrics such as CPU, network, and ephemeral-disk usage; uses Prometheus metrics and alerting; and maintains capacity headroom.
• Schema changes: Schema evolution is handled gradually, like API versioning. Producers announce a new version, continue supporting the old version during migration, and monitor remaining consumers. The speaker emphasizes that this also requires coordination between teams.

Format and scope

This is an engineering case study with an audience Q&A, rather than a product review or step-by-step tutorial. The technical explanations cover Kafka-compatible architecture, log compaction, recovery, mirroring, consumer tuning, and schema migration.

Speakers and sources

Pradeep Khileri of Blinkit/Eternal delivers the talk and answers questions. Several audience members ask questions; their affiliations are sometimes audible, including IFS and Thoughtworks. One attendee identifies himself as Rohit.