Stanford CS146S: How always-on agents work

Summary

Lee presents the architecture behind always-on, proactive agents, using Grok Bot as the main example. Unlike a terminal agent that stops when a laptop closes, an always-on agent runs in the cloud, maintains its own computer and files, preserves state across restarts, and responds to events such as Slack messages, emails, or scheduled routines—not just direct prompts. The goal is for agents to handle longer-running work in the background and notify users when there is something worth reporting.

How the System Is Structured

• Persistent server-side agent: The agent’s conversation, memory, routines, and other state are stored so the service can resume after interruptions. The server can initialize the agent when it receives a message or other trigger.
• Cloud computer: Each agent can use an isolated Linux environment running on a Firecracker virtual machine. It can access files, run commands, use a browser, and interact with a desktop. The machine can be started when needed, snapshotted, and shut down to conserve resources.
• Event routing and notification: Incoming events are routed, deduplicated, and added to the appropriate conversation. Heuristics determine whether an update is important enough to interrupt the user. Messages can be prioritized, and several arriving close together can be batched.
• Durable workflows: Lee distinguishes ordinary job queues from workflows that can resume from the point of failure rather than repeating completed steps. The system uses the open-source workflow tool Temporal for this kind of reliability.
• Thin client, thick server: Clients—such as mobile, desktop, email, or Slack interfaces—communicate with the server through a small set of interactions centered on “send to user.” The server handles the heavier agent work, while the client can provide interactive controls, such as reviewing an email draft or completing a login or payment flow.

Tools, Memory, and Context

• Files as persistent memory: User preferences, work journals, skills, and task notes can be stored in files. The agent can search them later to recall past work or follow a documented procedure.
• Skills and routines: A skill can encode a repeatable process in a Markdown file. Plugins and connectors can provide additional skills or access to services, while routines can run on a schedule.
• Dynamic tool discovery: Loading every tool definition into the model’s context can crowd out the conversation and increase cost. A “tool finder” approach keeps the initial context lean and lets the model find more detailed tool information when needed.
• Escalating information retrieval: The agent should try the simplest reliable source first: conversation context, then files or a service/API, then web search, then browser or desktop interaction. It should ask the user only if those options fail.
• Subagents: A main agent can delegate token-intensive or specialized work to separate assistants. They return results without filling the main agent’s context with every intermediate tool call. This gives the system the appearance of very large context, though models still have finite limits.
• Context and cache efficiency: Long outputs can be saved to files, expensive tasks delegated, and unneeded context pruned or summarized. Keeping prompts and tool definitions stable can preserve prompt caching and reduce cost. Lee also describes compressing long conversations while a provider’s cache is still warm, though preserving important details during summarization remains difficult.

Reliability, Security, and Product Development

The lecture emphasizes that extended autonomy needs stronger safeguards. Examples include checking commands before execution, treating emails and webhooks as untrusted inputs to reduce prompt-injection risk, and requesting user approval for irreversible actions such as sending messages. The product can also keep sensitive information, such as passwords, out of model conversations.

Lee describes a continuous improvement cycle: record agent errors, fix prompts or systems, evaluate whether the fixes work, and eventually train or deploy improved models. He notes that group-chat and multi-agent behavior introduce new challenges, and that evaluation becomes harder as agents are expected to work for weeks or months. Other open problems include memory quality, security, and deciding when proactive notifications are helpful rather than distracting.

Key Conclusions and Predictions

Lee argues that better models have made the user interface simpler: instead of approving every command, users can direct an agent conversationally while background systems handle more of the work. He predicts longer-running agents, stronger memory and skill-building, simpler interfaces, and more voice-based interaction.

He also recommends designing flexible products that can adapt as model capabilities change, and encourages engineers to understand both system architecture and how LLMs behave.

Guides, Tutorials, and Reviews

• Guide/tutorial: The talk is a technical overview of how to design an always-on agent, covering cloud execution, workflows, memory, tools, subagents, context management, security, and evaluation.
• Implementation reference: Temporal is recommended as an open-source option for durable workflows.
• Product review: No formal product review is presented; Grok Bot is discussed as an example of the architecture.

Main Speaker and Source

Lee is the sole speaker, presenting an engineering lecture and describing systems and practices associated with work at SpaceX and Cursor.