WWDC26: LLM search using Core Spotlight | Apple

Overview: LLM-driven search with Core Spotlight (WWDC26)

• The session introduces LLM search directly over an app’s Core Spotlight index using Foundation Models and a new tool.
• Goal: enable conversational, grounded answers where the LLM can reason and respond using only the app content you’ve made searchable.

Core idea: Tool-calling into Core Spotlight

The system uses the Foundation Models “tool protocol”:

• A tool declares:
  • Arguments
  • Output
  • Instructions
• The model decides when to call the tool, generates the tool arguments, then uses the tool output to produce the final response.

New product feature: SpotlightSearchTool

• Lets a language model search your app’s Core Spotlight content for contextual response generation.
• Available on iOS, iPadOS, macOS, and visionOS.

App requirements: donate/index searchable content

Before using SpotlightSearchTool, the app must donate searchable items to Core Spotlight.

• References a prior session on:
  • Supporting semantic search with Core Spotlight
  • How to donate content
  • How to manage donations via a delegate and reindex extension
  • Structured search over item attributes and semantic index search

Example scenario: “Hiking trails” app

The app indexes:

• Trail metadata (e.g., name, location)
• User data (e.g., completion date and personal notes)

Example user questions:

• “What hikes have I gone on?” → the model must search by attributes (date/location).

• “I remember being really happy on some of my hikes. Which ones were they?” → may require additional computation/logic.

How to adopt SpotlightSearchTool (three main steps)

1. Configure the tool

   • Import CoreSpotlight and FoundationModels.
   • Tool is ready to search with “one line of code” (per the session).
   • Supports custom configuration, such as searching file paths in the app sandbox (e.g., FileSource).
   • Choose model via Model Provider APIs (e.g., SystemLanguageModel or custom).

2. Provide additional context during search (to improve answers)

   • Constraint: some Spotlight metadata (e.g., text/HTML) is stored in a compact/searchable-only form that may not be directly readable by the LLM.
   • For better LLM output, provide additional model-readable metadata during search execution.
   • Enhances performance and scale by:
     • Extending the Core Spotlight index delegate with a method to recover the full CSSearchableItem by identifier.
     • Enabling efficient response handling even with millions of results.

3. Display results in the UI

   • The LLM session response provides a concise description of the result set (good for assistant-style chat UI).
   • SpotlightSearchTool can also provide search results directly for list-style UI:
     • Results arrive as async batched search replies.
     • Uses a query token so the UI knows when to refresh.
   • The model may call the tool multiple times before giving the final answer, so the UI should track reply tokens appropriately.

Spotlight search capabilities and customization

Built-in capabilities

• Semantic search over text
• Structured search over metadata like dates, people, locations, etc.

Guidance profiles (scope what the model searches)

• Guidance profiles constrain the tool’s capabilities to what your app content supports.
• Can also:
  • Specify the exact metadata attributes the model should use.
  • Set dynamic guide levels depending on the model and context size.
• Example rationale:
  • If the hiking app doesn’t include person relationships, guidance for authors/recipients can be skipped.

Reference resolution (map user mentions to indexed entities)

• If the prompt uses a reference (“that person”), the app can supply a resolver:
  • A contact resolver returns contact info matching the user’s identity.
• The tool then filters Spotlight results to the correct person entity in the index.

Custom pipeline stages (compute over result sets)

For complex queries, the tool supports pipeline search:

• Combines index queries plus computation over results.

• Example: “How many trails have I hiked this year, and for each month, how many miles on average?”

  • Model can request multiple pipeline stages (e.g., count by month → compute averages).

Custom pipeline stages

• Implemented by the app by registering stages in the tool configuration.
• Stages operate on CSSearchableItem inputs and return scored/derived outputs.
• Generable: the model can generate them on-demand.
• Stages may return partial outputs/data back to the app when useful.

Example custom stage: “happiness score”

• Compute a happiness score over each hike’s notes (e.g., via sentiment analysis or custom logic such as star ratings).
• Stage outputs scored results so the model can respond using top-scoring items only.
• Uses Guides to inform which results to prefer.

UI-friendly partial outputs

• Pipeline stages can return diverse data types:
  • Aggregate counts/tables
  • Free-form text
  • Computed numeric values
• Each reply can include an LLM-generated label to help the UI present results flexibly.

Evaluating the system (Evals framework)

The session discusses the Evaluations framework to verify:

• Whether the model calls the tool correctly
• Whether the response is meaningful and grounded

It also supports rapid iteration on:

• Search content donation
• Different guidance profiles for SpotlightSearchTool

Focused evaluation metric (example): result coverage

• Measures whether the model-generated answers include the expected items returned from Core Spotlight.

Evaluation setup in the hiking app example

• Define datasets using ModelSampleProtocol:
  • Each sample includes:
    • Natural language request (“TrailRequest natural language input”)
    • Expected response trajectory (including tool-calling expectations)
    • Set of expected search result item identifiers
• Seed samples can be expanded via Sample Generation APIs to broaden query variations.

Evaluation flow:

1. Load trail items and samples in the test target
2. Donate trail items to Core Spotlight
3. Configure SpotlightSearchTool for evaluation
4. Run evaluation and assert metrics like result coverage

Key takeaway

The session emphasizes shifting from writing search queries manually to:

• Providing indexed content
• Letting LLMs call SpotlightSearchTool and use tool outputs + pipeline computations
• Validating quality using the Evaluations framework

Main speaker / source

• Jennifer — Spotlight engineering team (Apple WWDC session).