End-to-end Exactly-once Aggregation Over Ad Streams | Yelp

Summary

Yelp’s local advertising platform processes events such as ad views, clicks, calls, and check-ins. Its streaming pipelines support campaign reporting and operational decisions, including tracking campaign trends, setting bids, and targeting. The talk focuses on aggregating these events throughout the day while keeping results accurate despite pipeline failures.

Requirements and Rejected Approaches

• Tumbling daily windows can aggregate a full day’s events, but generally produce a result only when the window closes. This does not meet the need to show partial totals as the day progresses.
• Cassandra counter increments would let the database accumulate updates, but the speaker says counters are not idempotent and can produce discrepancies. He cites reports of discrepancies as high as 5% in some industry use, depending on usage.
• Recomputing and writing totals in the pipeline provides more control over aggregation, but creates a retry problem: if a batch partially writes to Cassandra and the application fails before committing its Kafka offsets, replaying the batch can count already-written events again.

Exactly-Once Strategy

The proposed design uses Kafka’s ordered partitions and offsets to track which input data has already contributed to each aggregate:

1. Spark Streaming processes Kafka records in micro-batches, while the application manages offset commits.
2. For each aggregate key, such as a campaign ID and day, the pipeline reads the stored aggregate and its processing version from Cassandra.
3. The version records progress through the relevant Kafka partition—specifically, the highest offset processed for that key. With multiple partitions, the system tracks partition and offset information per key.
4. On replay, the pipeline discards events at or below the stored version and aggregates only newer events.
5. The updated aggregate and version are written back. If a failure causes a batch to replay, already-applied data is skipped. If a write did not succeed, the older version allows the data to be processed on retry.

This makes the aggregation logic idempotent across retries and addresses both partial-write failures and failures before offset commits. The approach is not limited to simple counts: the speaker says it can support more complex aggregation functions and richer data structures.

Further Work and Discussion

• Window joins: The team was exploring joins as a form of aggregation, allowing custom business rules—for example, deciding whether to emit a result when only some input streams contain matching data.
• Deduplication: The team was also working on handling duplicate input events, which can arise when upstream publishers are not idempotent.
• Late or out-of-order data: In the Q&A, the speaker described using a cutoff or window boundary at ingestion. Data arriving after the system has moved on from a period, such as the previous day, may be filtered out rather than added to that old aggregate.
• Spark Streaming and Flink: The speaker’s team built its system on Spark Streaming before Flink was mature enough to be considered. He noted that managing state and its lifecycle remained an important production concern, sometimes requiring work outside the frameworks.

This is a technical presentation rather than a product review or step-by-step tutorial. It walks through the requirements, alternatives, failure modes, and design of the aggregation system.

Main speaker/source: Amit of Yelp. He says a co-presenter could not attend; the name is unclear in the auto-generated captions. Audience members contribute questions in the closing Q&A.