Video summary

Club TWiT: AI User Group #16 - Building Better Agents

Main summary

Key takeaways

Technology

Summary: AI User Group #16 — “Building Better Agents” (Club TWiT)

What the group focused on

The discussion centered on building robust AI agents—not just chatbots. Agents become truly useful when they support:

  • Delegation / orchestration: choosing tools and models for subtasks
  • Memory systems: multiple “memory hooks” and support for long-running context
  • Structured workflows inspired by SDLC/DevOps (e.g., specs → execution → QA → PR → iterate)
  • Portability across agent frameworks, so work can move as tools evolve

Key product / technology concepts mentioned

1) Hermes agents: robust memory + delegation + tool selection

Speakers described using Hermes to build “better agents,” emphasizing:

  • Stronger memory hooks via multiple parallel memory tools
    • Basic memory (“Memory MD”, small by default, configurable)
    • Additional memory systems such as LLM wiki, “hindsight,” and a SQL-lite + semantic search approach
    • Ability to run multiple memory systems simultaneously with low context overhead
  • Delegation
    • Hermes can select specialized tools/models per task, for example:
      • Gemini for images
      • Groq for voice
      • Claude / Claude Code for coding tasks
      • (Examples varied by speaker)
  • Portability via OpenAI-compatible APIs
    • Claim: migrating from OpenClaw to Hermes is easier because both align with OpenAI API standards, reducing vendor lock-in

2) Profiles as “roles” for SDLC / DevOps workflows

Hermes profiles were used to mirror real team roles:

  • One profile for spec-writing
  • One agent to execute issues (turn specs into actions/tasks)
  • A QA agent to validate and generate PRs

A key emphasis: consistent outcomes require good specs and agent instruction files. Vague specs lead to inconsistent outputs.

3) Scheduling and automation via “cron jobs” (systemd underneath)

Hermes’ web UI showed cron-like jobs, described as:

  • Not “crown jobs” (i.e., beyond basic cron)
  • Implemented using systemd, but configurable with cron syntax

Example use case:

  • Weekly benchmarking of models and updating delegation behavior based on results

4) Web UI vs desktop app, remote operation, and local vs server execution

Main operating style discussed:

  • Hermes was presented primarily through a web-based UI
    • Chosen for usability (editing prompts, browsing sessions, attachments)
  • Remote access concept:
    • Can run remotely via TailScale (phone/tablet access without installing)
  • Desktop considerations:
    • Desktop might enable UI automation (screen/mouse control), which a web UI can’t easily do
  • Strong preference for a central server + agent running, managed via SSH / TailScale
    • (A “centralized administration” approach)

5) “BTW” / side commands and conversation threading

Hermes included features like BTW to create/branch a related task/issue within the same conversation without mixing contexts.

There was also mention of steerq/beta-style “side commands” as workflow helpers (subtitles were unclear, but the “side commands” idea was stressed).

6) Web search tooling: SearX / Brave / Kagi + “Pulse” substrate

Speakers compared approaches for web search and how results are wrapped for agents.

  • SearX / “Searching” concept
    • Queries multiple sources and summarizes into a consistent, JSON/API-friendly output
  • Pulse
    • A local skill/service aggregating from many sources (examples mentioned):
      • Reddit, Hacker News, YouTube archive, RSS, GitHub
      • academic/news sources, etc.
    • Claimed benefit: reduces token costs by pre-processing / extracting data rather than feeding raw HTML to LLMs
  • Pricing preferences debated:
    • Some prefer predictable billing (fixed fees) instead of surprises from per-token costs
    • Brave and Kagi pricing were discussed as “worth it” depending on usage frequency

7) Security / privacy debate: local models vs cloud logs

A major theme was privacy and security risks with cloud-based agent platforms.

Concerns included that cloud agents might send:

  • chat sessions/transcripts
  • logs
  • summaries
  • possibly memory/agent state

Counterpoint discussed:

  • Interest in local models and local agents to keep data local
  • At least one speaker claimed they can turn off memory systems, but acknowledged:
    • the platform may still send session info
    • logs likely exist for troubleshooting
  • Incentive argument:
    • companies are unlikely to leak secrets due to business risk
    • but the possibility of subpoena/log access remains

Reviews, benchmarks, and “coding agents” analysis

Coding model comparisons and delegation “where the magic is coming from”

Speakers referenced benchmarks such as the Deep SWE benchmark and described side-by-side comparisons, including setups akin to DeepSeek / Claude / GitHub Copilot.

Differences mentioned included:

  • handling both synchronous and asynchronous requirements
  • adherence to security principles

A recurring point:

  • It’s often hard to know whether performance comes from:
    • the model
    • or the agent harness / orchestration

Practical recommendation:

  • Use a good agent instruction file (e.g., agent.md / system spec)
  • Use good requirements/specification docs
  • Avoid overly large or random instruction files

Tutorials / workflows shared (practical takeaways)

Example SDLC agent loop (suggested pattern)

A suggested workflow pattern:

  1. Create an “architecture/spec” markdown file (or files)
  2. Use Hermes profiles/agents to: 1) Generate specs / tickets 2) Execute tasks 3) Run QA, verify results, and produce PRs 4) Iterate

“Architecture.md” / design documentation for reusable structure

Speakers recommended maintaining consistent markdown architecture so outputs follow a stable system design:

  • Prevents random code generation
  • Improves maintainability for developers
  • Helps support migration across tools/platforms if the “architecture contract” remains consistent

Example: AI-assisted DevOps / migrations with gates / checkpoints

One speaker described agent-driven automation with steps such as:

  • “Pass gate 0 before proceeding”
  • produce verification/documentation output
  • use TPM/password storage to reduce manual intervention
  • generate full procedural documentation

Hardware / local compute notes (agents need fast, cheap cycles)

Notes included running multiple large models locally and potentially using clusters.

  • Example mentioned: “dual GPUs” (two 5090s / Blackwell variants referenced)
  • Theme: local compute can reduce per-token costs, but the upfront cost remains high
  • A “cloud vs local” emotional tradeoff was noted:
    • guilt about paying cloud costs vs “wasting” idle local capacity

Other notable discussion: agentic futures and converging UIs

The group observed multiple products converging on similar UX patterns:

  • desktop/web interfaces
  • profiles/roles
  • agentic modes

Broader claim:

  • agents are moving toward:
    • always-on background capability
    • continuous optimization (cost/speed)
    • integration into normal consumer experiences (voice, always-ready assistants)

Speakers / sources (main voices referenced)

  • Leo Laporte (host / “This is Twit” intro and wrap-up)
  • Darren Oki (guest; described running/using Hermes and workflows)
  • Larry Gold
  • Error 404 in town (Bill mentioned; exact role/source unclear from subtitles)
  • Craig (newer attendee; CTO of a large IT transformation software company)
  • Anthony (referenced multiple times; contributions on dev/agent tooling)
  • Patrick (sales/operations “one sheet” skill work; Slack integration mentioned)
  • Jeff Atwood (mentioned for scheduling/podcast show planning)
  • Lisa (used Artemis/Hermes profiles; provided feedback/requirements in examples)

External entities/models/tools mentioned: Hermes, OpenClaw, Claude Code, GitHub Copilot (GPxx), DeepSeek, Groq, Gemini, Firecrawl, SearX, Brave, Kagi, Pulse, Bitwarden, Nexcloud.

Original video