Video summary

AWS re:Invent 2025 - Harnessing analytics for humans and AI (INV201)

Main summary

Key takeaways

Technology

Summary of the video’s technological concepts and product/analytics analysis

Emerging trend 1: Agentic AI for analytics across the full data journey

  • Agentic AI + faster analytics workflows

    • AWS positions AI agents as helpers throughout the data lifecycle—writing code, processing data/pipelines, and working with data products.
    • Emphasis is on speed advantages over human-driven iteration.

  • AI-integrated upgrade and migration for Apache Spark

    • Optimized Spark 3.5.6 engine launched across multiple AWS services, including:
      • EMR
      • Glue
      • Athena (Spark/engine usage)
      • SageMaker notebooks (and “Spark Connect” usage mentioned in the demo)
    • Performance focus on Iceberg
      • Improved read/write performance on Iceberg tables
      • Faster interactive querying in Athena
    • “Spark upgrade agent” (AI assistance for migration)
      • Built on a knowledge base compiled from thousands of Spark upgrade outcomes (successful and failed)
      • Capabilities include:
        • Planning the upgrade
        • Reading error messages
        • Automatically remediating failures
      • Designed to reduce Spark upgrade effort from months to weeks (claimed)
      • Covers upgrades from older versions, such as:
        • Glue 2.0
        • EMR on EC2 5.2
        • prior EMR Serverless versions
    • Example customer
      • FINRA plans to anchor Spark version management on the upgrade agent in 2026

  • AI-assisted development in SageMaker notebooks

    • AWS introduced a new SageMaker notebook experience:
      • Serverless notebook (no cluster/infrastructure tuning)
      • Polyglot support in one notebook:
        • Python + SQL together
        • PySpark via Spark Connect
        • SQL querying across Redshift, Athena/Trino, and Snowflake
      • Natural language objectives
        • Users describe goals in plain language
        • The agent uses context from data catalogs/metadata to generate plans and code
      • End-to-end workflow coverage
        • querying
        • exploratory analysis
        • ETL
        • ML training
        • visualization generation
    • Built-in agent features showcased in the demo
      • Auto-planning and execution
        • Agent reads NYC taxi Iceberg data from S3 into a Spark dataframe and summarizes schema/statistics
      • “Fix with AI” for exceptions
        • Analyzes stack traces/visualization failures
        • Generates corrected code for quick acceptance and rerun
      • Data cleaning with generated transformations
        • Includes visual evidence of improved quality
      • ML model training assistance
        • Example: regression for trip prices
        • Includes feature importance, accuracy metrics, and prediction visualization code
      • Insight-to-visualization
        • Ask questions like “key drivers of price”
        • Produces both:
          • text findings
          • code for dashboard-ready visualizations
      • Writing results back to the lake
        • Agent generates Spark code to write cleaned data back as Iceberg tables in the data lake

Emerging trend 2: Open analytics architectures and composable “building blocks”

  • Why open architectures?

    • Customers want:
      • flexibility (avoid “one API fits all”)
      • freedom to choose analytics tools by workload
      • avoidance of expensive data migration

  • Open data formats as the foundation

    • AWS emphasizes Iceberg (and related open formats like parquet/iceberg)
    • Goal: enable swapping analytic engines while keeping data consistent

  • “Analytics building blocks” concept

    • AWS describes disaggregating core analytics primitives from specific services so they can be reused across multiple AWS tools and environments

  • New cross-service building block: Materialized Views for Iceberg (fully managed)

    • Introduced capability:
      • Created using the Spark 3.5.6 engine
      • Stored as Iceberg tables in S3 with Iceberg/Glue Catalog integration
      • Automatically maintained as data changes (no manual orchestration)
    • Appears as a regular table in Glue Data Catalog
    • Queryable by:
      • Athena
      • SageMaker Unified Studio / notebooks
      • VS Code / JupyterLab
      • Redshift
      • and other Iceberg-compliant third-party engines
    • AWS claims:
      • query rewrite automatically uses the views for performance
      • up to 8x performance improvements in testing without code changes

  • Another example building block: S3 Tables

    • Cross-engine accessible tables for any Iceberg-compliant analytics engine
    • AWS highlights rapid iteration and expanded features (e.g., intelligent tiering and cross-region/cross-account replication)

  • Customer scale example (Intuit)

    • Intuit’s strategy aims to reduce “time to discover and access data”:
      • ~20 days → 14.2 hours → 9.4 hours, with further reduction targets
    • Key practices include:
      • Standardizing data semantics via a semantic layer
        • domains/subdomains (e.g., customer/business)
      • Standardizing infrastructure/pathways
        • pipelines for streaming/batch
        • product events / third-party data
      • Metadata/ownership/lineage as a core enabler for productivity and trust
      • Quality/operational requirements across dimensions, including:
        • stewardship documentation
        • data model
        • observability
        • operational stability
    • Demoed GenAI/agentic analytics
      • Intuit assistant (“Intuit Assist”):
        • semantic search + metadata
        • answers questions and produces results suitable for notebooks
    • AWS services supporting Intuit workloads:
      • transactional: Aurora, Dynamo
      • streaming/batch and cataloging: EMR, MSK, Lake Formation, Glue Data Catalog, S3
      • analytics/dev tools: SageMaker, Athena, QuickSight
    • Architecture style described as “zero-ETL”
      • data moves, but downstream teams see near-instant availability

Open formats + performance improvements for Iceberg across AWS analytics

  • AWS highlights investments in Iceberg support across:

    • the data layer (S3 Tables)
    • engines/services such as Athena and Redshift
    • broader engine ecosystem including EMR and more

  • Redshift Iceberg performance and new write support

    • AWS claims 2x improved Redshift read performance on Iceberg via:
      • distributed bloom filters
      • metadata caching
      • optimized query planning
    • Announced at re:Invent:
      • Iceberg table append write support in Redshift
      • shift toward writes, not only read/query optimization

Open warehouse architecture and Iceberg as the “universal language” (Supabase + Snapchat)

  • Supabase (CEO Paul Copplestone)

    • Argues database selection must be forward-looking, since moving data later is costly
    • Notes common issues when teams start with Postgres for analytical data and later add a second database/warehouse:
      • duplication
      • fragmentation
      • increased complexity and cost
    • Open Warehouse Architecture
      • Keep Postgres for low-latency app queries
      • Store analytics data once in S3 Tables (Iceberg) as the unified storage layer
      • Maintain sync automatically between Postgres and Iceberg so data appears in S3 Tables without manual migration
    • Cost optimization goal:
      • avoid moving data into multiple warehouses
      • connect engines directly to S3/Iceberg
    • Mentions S3 Vectors (embeddings)
      • extending the architecture beyond analytics into semantic/AI features

  • Snapchat (Terek; Spectacles/Lens developers)

    • Uses Supabase/S3/Iceberg-based access to enable more complex AR lens development
    • Notes Supabase capabilities including:
      • WebSockets for real-time connections
      • Postgres extensions for querying Iceberg files

  • AWS vectors focus (semantic understanding / agent memory)

    • AWS describes S3 Vectors (GA) with scale/performance claims:
      • max vectors per index increased to 2 billion
      • up to 10,000 indexes per bucket (potentially up to 20 trillion vectors per bucket)
      • high throughput and low warm-query latency
    • Use cases:
      • semantic metadata/media understanding
      • hybrid search
      • agent memory extension by storing contextual vectors (e.g., user/account/interaction context)

Main speakers / sources

  • Mai-Lan Tomsen Bekovec — Vice President, Technology, Data and Analytics, AWS (main presenter)
  • Daiyan Alamgir — Principal Engineer, SageMaker Unified Studio
  • Tristan Baker — Distinguished Engineer, Intuit
  • Terek — Product Manager, Spectacles (Snap Inc., via Supabase/AWS)
  • Paul Copplestone — CEO & co-founder, Supabase
  • Matt Garman — referenced for the GA announcement of S3 Vectors

Original video