Distill the LLM, Don't Serve It: Search & Personalization at DoorDash — Raghav Saboo, DoorDash

Technological concepts / system approach (DoorDash search & personalization with LLMs)

Core thesis: semantic understanding is the bottleneck for discovery

• Effective marketplace “discovery” depends on understanding:
  • what items mean in context, and
  • what the shopper intends,
  • not just optimizing for engagement/clicks.
• DoorDash has expanded beyond restaurants (e.g., grocery, retail, pets, gifts), where:
  • user goals are often broad (e.g., “I just got a puppy…”),
  • goals evolve over multiple sessions/days,
  • creating a long “shopping arc” that is difficult to model end-to-end.

LLM integration via “four primitives”

The talk frames LLM usage as producing shared representations and offline reasoning outputs that plug into existing production search/ranking/content pipelines—rather than replacing those systems with a full online LLM.

---

Primitive 1: Supervision (teach search/ranking what “good” means)

Problem

• Ranking by engagement alone can surface popular but irrelevant items.
• Example: someone searches for gluten-free pasta; popular regular pasta may rank higher, and even unrelated gluten-adjacent items can appear.

Two sources of truth (and their limitations)

• Human labels
  • Expensive/slow
  • Can become outdated due to fast catalog changes
• Behavioral signals
  • Large, but biased by:
    • exposure/position
    • price/promotions
    • prior model selection

LLM-based “scaled reasoning” to create a high-quality dataset

• Start with human “fundamental truths” and generate examples of:
  • query element ↔ product element pairs
  • a graded relevance label (0–2)
• Use suspicious-case checks:
  • if human labels conflict with strong add-to-cart/conversion performance,
  • re-evaluate using a “stronger LLM” for guidance
• Align relevance labels with other model outputs (e.g., taxonomy/category validity) to correct/adjust labels
• Result: a high-precision golden dataset

Model distillation / reuse

• Fine-tune a lightweight model (example mentioned: GPT-4o mini) on the golden labels.
• Use the fine-tuned model as an offline tagger to generate relevance/evaluation tags across the catalog.
• These tags become a common target for both search and ranking systems.

---

Primitive 2: Catalog semantics (learn a comparable, scalable representation of products)

Problem

• DoorDash has a catalog that can reach billions of store items globally.
• SKUs are unique but not semantically informative.
• Human taxonomy is helpful but can be too coarse/rigid:
  • e.g., “hot sauces” may not capture relationships among brands/types (Huy Fong, Frank’s, Tabasco).

Solution: semantic identifiers / learned taxonomy code

• Introduce short hierarchical codes (a learned taxonomy) with controllable fine-grainedness.
• Codes support:
  • Prefix-based similarity (nearby codes imply semantic closeness)
  • Cross-model comparability (usable by both LLM and traditional retrieval/ranking models)

Benefits described

• Inter-category connections
  • Chips/salsa/guacamole may live in different taxonomy branches but share semantic neighborhoods.
• Cold start handling
  • New items can be added using n-gram / byte-pair encoding into sparse identifiers so they inherit signals.
• Tail coverage
  • Sparse items inherit signals from semantically related items.
• Reverse auditing
  • Test how well learned semantics match human labels.

Measured impacts (examples)

• Major ranker improvement: MRR up ~4–5%, translating to conversion gains.
• Better query reformulation (example intents):
  • Sriracha → garlic-chili, sambal oelek
• Mentioned output insight:
  • query graphs can lose semantics when inventory is missing,
  • semantic identifiers help recover improvements.

---

Primitive 3: Semantic personalization via “memory”

Question addressed

• How does the system represent consumer context?
• Can that knowledge be reused across multiple models?

Approach: memory blocks in multiple time scales

The system builds user memory in structured forms so it can be reused by:

• search/ranking models
• personalization modules
• agentic experiences (e.g., “Ask DoorDash”)

Three time scales

1. Long-term memory
   • enduring preferences from orders/search/browsing/support interactions
2. Real-time context
   • within-session/agent interactions (cart status, active searches)
3. Stated preferences
   • explicit constraints given through agent interactions (e.g., restrictions and preferences)

Memory representation formats

• Human-readable text summarizing the consumer compactly
• Latent vectors (embeddings of memory blocks) for search/ranking
• Graph/tree-like structures capturing associations:
  • users ↔ brands
  • users ↔ taxonomy concepts
  • revealed preferences

Example mechanism

• Build context graphs that retrieve memory concepts.
• Graphs must fit the “shopping path,” where interactions can be sparse and multi-hop.
• Claim: graph-based embeddings can outperform taxonomy-based embeddings at fine-grained levels.

Where memory is used

1. personalized collections (store pages)
2. agent-based personalization (Ask DoorDash)
3. passed into search/ranking as functions/input components

---

Primitive 4: Managed content generation (LLM-controlled, offline generation for UI surfaces)

Goal

Generate content forms (e.g., ranked lists, carousel headings, sub-copy) using:

• semantic identifiers
• memory blocks
• graded relevance / LLM-controlled reasoning

Product feature: control & manageability

• Content can be generated/updated in response to events and moments for different consumers.
• Generation is offline/batched:
  • LLM produces outputs
  • existing search/ranking stack handles hydration/ranking
• This keeps production manageable and aligned with business logic.

Concrete example: personalized collections

• Historically, collections were a fixed library/attribute-based.
• Now extended to consumer-level generated collections:
  • Offline LLM synthesizes collections from consumer memory + semantic identifiers
  • Produces headings, subtitles, and item sets
  • Example personalization logic:
    • plant lovers see plant collections
    • cat-only households see cat food
    • restocking leads to more staples

Early results

• Pet vertical:
  • ~+1% orders
  • +0.6% active users in early tests

---

How LLM reasoning is “moved into” production search/ranking (not full LLM replacement)

The talk emphasizes a pattern:

• Don’t replace the entire search/ranking system with LLMs.
• Instead, move reasoning and understanding into specific production ranking architectures.
• Use LLMs for offline labeling/tagging and model training, while smaller faster models serve online needs.

Search training method described (two-step comparison)

• Standard embeddings can blur relevance distinctions at ecommerce scale.
• Solution: a two-stage comparison method trained using graduated relevance labels:

1. Extraction stage
   • two-tower encoders
   • multi-level controlled comparison losses
   • generating complex negatives
2. Curriculum stage
   • re-train using complex negatives
   • sharpen separation between relevant vs moderately relevant

Reported outcome:

• NDCG improved ~2.3%, with downstream business metric improvements.

Ranking architecture described

• Relevance labels derived from LLM-guided supervision are integrated into ranking models.
• Add an ordinal relevance tower alongside engagement towers (click/add-to-cart/conversion).
• Because towers share layers:
  • semantic matching and relevance matching can be predicted in one pass
  • relevance probabilities can be combined via a value function to balance engagement vs relevance across surfaces.

---

Main speakers / sources

• Raghav Saboo, Full-time Machine Learning Engineer at DoorDash (presenter).