We Saved Millions (Really!): Marc Brooker & Carr Onstott - Strands Decider Standup

Summary

The episode explains how Strands Decider, a compact “System 1” decision model, can handle bounded classification and routing tasks faster and more cheaply than using a general-purpose LLM for every request. The speakers discuss its development, a production deployment for Amazon’s internal ticketing system, and ways to evaluate and deploy it.

How Decider Works and Why It Matters

• Decider sits between conventional deterministic code and more open-ended LLMs: it interprets language but returns a constrained decision that software can use directly.
• For tasks such as classification, it can replace an LLM call that may generate many tokens with a short, single-token decision, reducing latency and token costs.
• The team describes it as a practical middle ground: more flexible than a hand-built classifier, but faster and less expensive than using a large LLM for every decision.
• Marc Brooker said the initial model began as an experiment on a single RTX 3090, using a roughly 2-billion-parameter model. He worked with coding agents to explore architectures and training methods. The original goal was to learn, rather than necessarily to build a production-ready model.
• Fabio reported that V2 was in training, with more than 12,000 GPU hours accumulated at the time of the recording.

Production Case: Internal Ticket Classification

Carr Onstott’s team uses a classifier to assign incoming tickets to around 20 labels, including test, security, availability, latency, and scaling issues.

The rollout proceeded in stages:

1. The team tried Decider on a sample of recent tickets.
2. They deployed it alongside the existing Sonnet model in shadow mode, initially on a portion of traffic.
3. They compared the results. The models agreed on about 76% of tickets. The team investigated disagreements using separate LLM-based evaluations. In one blind comparison, the evaluator favored Decider about 5% more often; in an A/B comparison, the results were roughly even.
4. They expanded shadow traffic to all tickets and checked system stability before switching over.

The team concluded that Decider was no less accurate for its use case and moved it into production. Carr said the effort took roughly two and a half weeks, from the first setup commit to switching off the old model.

After the change, the team saw Bedrock token use and Lambda duration fall substantially. Concurrent Lambda invocations also dropped by about two-thirds. The speakers noted that shorter calls reduce both direct model costs and the time Lambda functions spend waiting for network responses. The blog post and discussion describe savings of millions of dollars, though the subtitles do not provide a precise figure.

Deployment and Further Uses

• The ticketing team used a SageMaker endpoint with automated scaling. The speakers said deployment recipes were available in the repository.
• Fabio announced a public Decider container image in ECR for SageMaker, so users would no longer need to build the image themselves.
• Carr said his team used the model out of the box. Their main experimentation involved trying different prompts for the same ticket, rather than retraining or fine-tuning the model.
• Potential follow-on uses include ranking ticket comments by usefulness during large incidents and replacing slower LLM-based inference elsewhere.
• Marc suggested that tools for automating model selection and evaluation—such as shadow testing and accuracy comparisons—could make adoption easier. He also highlighted hybrid workflows that combine fixed logic, decision models, and more capable agentic systems.

Review, Guide, and Tutorial Information

• Production evaluation guide: The team’s shadow-mode rollout, agreement checks, disagreement analysis, and staged move to full traffic provide a practical example of evaluating a replacement model before deployment.
• Deployment guidance: The discussion points viewers to SageMaker recipes in the repository and announces a public ECR image for SageMaker. The episode does not present a step-by-step tutorial.
• Related reading: The speakers reference Carr’s blog post about replacing Sonnet with Strands Decider in the ticketing system.

Main Speakers and Sources

• Mike Chambers — host
• Fabio Nonato de Paula — host and Strands Decider team member
• Marc Brooker — AWS distinguished engineer; discussed the model’s origins and future direction
• Carr Onstott — AWS principal engineer; described the internal ticketing deployment and its evaluation
• Source discussed: Carr Onstott’s blog post on using Strands Decider for internal ticket classification.