Video summary

Inside Apple Intelligence and Xcode: Special Presentation | WWDC26

Main summary

Key takeaways

Technology

Tech / Feature Summary (WWDC26: Apple Intelligence + Xcode)

1) Apple’s “integrated” AI platform concept

Apple frames AI as not a bolt-on layer, but an integrated stack spanning:

  • Silicon + OS + frameworks/models + developer tools
  • A focus on coherent reinforcement across layers, with privacy architecture, Swift APIs, and tooling designed to work together
  • Support for multiple developer personas:
    • Builders
    • Integrators
    • Product engineers
    • Researchers

2) Xcode 27: agentic coding for whole-project changes

Core promise: agentic coding is no longer just “chat”—agents in Xcode see the full project, plan work, and apply multi-file changes.

Key capabilities and UX

  • Agent conversations in the toolbar
    • Each conversation gets the full main editor area
    • Plus a dedicated pane for plans, files, and previews
  • Agents can:
    • Plan across the codebase (not just single-file edits)
    • Apply changes across files
    • Use tools such as:
      • Previews, playgrounds
      • Device integration / simulator
      • Localization
      • Debugging, testing

Rich context passed to the agent

Xcode provides additional context including:

  • Project configuration and build settings
  • What’s open/selected in the editor
  • The “meaning” of references (e.g., resolving what “this” points to)

Security/control model

  • Introduces a kernel-level file access permission model (macOS 27 feature implied)
  • If an agent needs file access:
    • The kernel asks Xcode first
    • Xcode enforces policies
    • Agents can only touch what Xcode allows

Demo outcomes (practical results)

  • Starting from a folder of pin images + a sketch concept, an Xcode agent:
    • Generates a SwiftUI app (“Pin collection by year”)
    • Uses an Asset Catalog for pin images
    • Chooses persistence (e.g., User Defaults first, Swift Data later)
    • Produces a working UI from the sketch
  • Follow-on agentic iterations:
    • Adds animations (sparkles, scale/spin, holographic 3D effect)
    • Adds a corkboard view with randomized pin placement
    • Runs parallel conversations, with developers able to switch and monitor progress

Accessibility/quality support

Mentioned support includes:

  • Adopting new APIs and localization
  • Localization flow with multiple stages and parallel subagents for translation

Model integration extensibility

Xcode supports agent/model workflows via:

  • Integrations for Anthropic/OpenAI/Google agents
  • Agent Client Protocol (ACP) to bring in any agent/model (including local Mac-run agents)

Plugins:

  • Installed via GitHub URL
  • Include:
    • “skills” (Markdown)
    • MCP tools (external tool connections)
    • Agent supply via Agent Client Protocol
  • Preserve developer control with approval gates for plugin access

3) System integration via App Intents (Siri + Apple Intelligence)

Speaker: Michael Gorbach (App Intents core team)

Goal

Enable an app’s content and actions to be accessible through Siri using:

  • Natural language
  • Contextual behaviors

What App Intents enables

  • Siri can search and reason over app content safely/private
  • Siri can act inside the app based on language (e.g., alarm/timer management)
  • Siri can use on-screen context so users can refer to what’s visible

Framework building blocks

  • Entities (AppEntity)
    • Represent app data items (e.g., a timer entity with properties like duration, label, state)
  • App Schemas
    • Define domain concepts (e.g., “timer” in the Clock domain)
    • Help Siri understand inputs/outputs consistently
  • Intents (AppIntent)
    • Represent actions (e.g., update timer)
    • Implemented as Swift structs with parameters/outputs

Semantic indexing and discovery

  • Uses a semantic index / datastore for entity search by meaning/property
  • Developers index entities via:
    • IndexedEntity protocol
    • indexAppEntities when entities change

On-screen awareness

  • Entity Annotations
    • SwiftUI modifiers / APIs to link UI views/selections to entities
  • Extends beyond UI to frameworks like:
    • notifications
    • Now Playing
    • AlarmKit so Siri can act even when the app isn’t foregrounded

Personalization learning

  • Interaction Donations
    • Developers donate what happened in-app (e.g., reset stopwatch)
    • Siri can learn preferences/route patterns and infer entity state transitions

Demo app

  • HotTickety
    • Alarms/timers/stopwatches controlled through Siri via natural language
    • Create/search/cancel/delete items

4) Foundation Models framework: unified Swift API for LLM features

Speakers: Louis (lead), later Matt (model plumbing demo), Rob (Evaluations)

Core idea

A native Swift API where:

  • Apps send prompts
  • Models return structured/unstructured responses

It goes beyond text generation to support:

  • Guided Generation (structured outputs)
  • Tool Calling
  • Instruction/tone control
  • Streaming output and real-time UI updates

Key updates announced

  1. Image Input (multimodal prompts)
    • Text + images
    • Includes a Generable macro to extract structured data from images
  2. Private Cloud Compute (PCC)
    • On-device model for fast/private/offline tasks
    • Server model on PCC for complex reasoning
    • Access for App Store Small Business program (<2M downloads)
    • Emphasis:
      • No account setup
      • No API keys
      • No token cost to developers
      • Daily per-user limits
      • Privacy promise (no storage; verifiable promise)
  3. Third-party model integration
    • Any provider can supply a Swift package conforming to LanguageModel
    • Examples mentioned: “Core AI models package” and “any LLM provider” via Swift packages
  4. Agentic experience APIs
    • Higher-level APIs for agentic workflows in-app
    • DynamicProfile for multiple sessions with shared context (reduces boilerplate)
  5. Testing via new Evaluations framework
    • Evaluations integrated into Xcode testing

Developer-facing model plumbing demo

Swapping models using the LanguageModel protocol:

  • Start with SystemLanguageModel
  • Replace with a Core AI language model (or others like Qwen) with minimal code changes

5) Evaluations framework: model output quality + “evaluation-driven development”

Speaker: Rob

Problem

LLM outputs are unpredictable, requiring systematic quality measurement.

What Evaluations provides

  • Define metrics and grade outputs
  • Metrics can be:
    • pass/fail
    • scoring (numeric)
  • Metrics can be aggregated across datasets to track statistical trends

Advanced evaluation patterns

  • Model judges using language models to approximate human judgment
  • ScoreDimensions for qualitative scoring (e.g., “relevance”, “usefulness”)

Workflow integration

  • Runs in test targets
  • Swift Testing integration via a new trait: evaluates
  • Xcode 27 adds Test report UI for evaluation results and metric breakdown
  • Supports iterative “hill-climbing”:
    • refine system instructions based on evaluation outcomes
  • Called Evaluation-driven Development

6) Core AI framework: on-device custom model execution (high performance)

Speakers: Raziel (Core AI lead) + Geppy (demo)

Core promise

On-device AI execution with:

  • Privacy (no cloud reliance)
  • No inference cost
  • Fast inference using Apple Silicon:
    • CPU / GPU / Neural Engine

Built for the full lifecycle:

  • creation, optimization, inference
  • fine-tuning
  • integration and debugging

API characteristics

  • Modern Swift API designed to progressively expose capability
  • Models loaded into app/device; run functions consume/produce NDArray

Tooling

  • Python tools with familiar PyTorch workflows
  • Converts/exports PyTorch models to Core AI format via Core AI TorchConverter
  • Stores converted models as assets for on-device execution

Customization and performance controls

  • Control model creation/optimization/execution
  • Advanced optimization knobs (including mention of custom kernels)
  • Execution performance controls:
    • cache management and resource purchase timing
    • pre-allocation of state/output buffers (important for LLMs)
    • async pipelines and parallel execution

Developer tools

  • Pre-compile models ahead of time
  • Core AI Debugger app:
    • visualize compute graphs
    • inspect tensors
    • trace back to Python sources

Demo highlights

  • iPad image + generative ideation tool (style retargeting) running fully on-device
  • Converted VLM + FLUX from PyTorch with a few code lines:
    • both models run locally
  • Scaling demo:
    • swap in larger variants on macOS with the same code
    • Core AI makes model swapping straightforward

Role in Apple Intelligence + Siri

“Powered by Core AI” is stated as the underlying inference framework for Apple’s most advanced on-device intelligence.

7) MLX: open-source Apple Silicon-first ML training/inference + distributed scaling

Speaker: Ronan; demos by Angelos and Yagil (LM Studio)

MLX pillars

  • Numerical computing library (vectorized ops / linear algebra)
  • Automatic differentiation
  • Distributed computing across Macs
  • Support for vision/speech/language models locally

Platforms / languages

  • Apple Silicon acceleration (Metal/GPU; mentions GPU Neural Accelerator support)
  • Supports:
    • Python
    • Swift
    • C/C++
  • Open source
    • available on GitHub and HuggingFace
    • 10k+ models supported out-of-box by MLX-LM

Distributed inference/training

  • Uses distributed computing with high-bandwidth/low-latency connectivity

Demo highlights

  • Angelos (M5 Max):
    • chat with an entire book using KV cache (very fast throughput)
    • generate cover art using a diffusion model (fully on-device)
  • Scaling with multiple machines:
    • speed up by distributing work across two MacBook Pros
    • uses Thunderbolt 5 RDMA for efficient interconnect
  • Frontier model cluster demo:
    • LM Studio + MLX Distributed
    • run a 1T parameter model (Kimi 2.6) distributed across 4 Mac Studios
    • model runs locally; real-time remote prompting via LM Link (encrypted remote access)
    • claims no token cost and no cloud data transfer for the described runs
    • distributed inference support in LM Studio “coming later this year”

Main speakers / sources (as stated)

  • Ken
  • Jerome
  • Michael Gorbach (App Intents core team)
  • Louis (Foundation Models / Evaluations lead-in)
  • Matt
  • Rob (Evaluations framework)
  • Rachel
  • Raziel (Core AI)
  • Geppy Parziale (Core AI demo)
  • Ronan (MLX)
  • Angelos
  • Yagil (LM Studio)

Original video