Video summary

You Can Learn Hermes Agent Harness In 20 Min | Loop Engineering, Memory System, Self-Improving AI

Main summary

Key takeaways

Technology

Summary of technological concepts & product features (Hermes Agent Harness)

  • Hermes agent harness (open source): Runs a self-improving LLM agent locally on a user machine (e.g., Mac/Windows). The speaker claims it’s a popular project with 200k+ GitHub stars.

Self-improving / built-in learning loop

  • The agent creates its own skills and improves over time based on experience (not purely user-written code).
  • Learning is persistently stored locally.

Ephemeral agent runs + persistent memory

  • Each request triggers an agent run (ephemeral), where the prompt plus chat context feeds the model.
  • After the run, relevant knowledge is saved into persistent local files / a database.

Interaction / gateway options

Interfaces

  1. WhatsApp gateway: Message in WhatsApp → agent runs → reply sent back to WhatsApp.
  2. Hermes desktop app: Direct UI for prompts (in addition to other integrations).

Local hosting

  • Because it runs locally, the system could be hosted on a virtual machine to “run forever” and answer WhatsApp questions.

System design (core flow)

Prompting pipeline

  • User prompt + current chat history + system prompt
    • → working memory
    • → LLM agent answers

Tool-using loop engine (loop engineering)

  • Hermes uses a loop engine to repeatedly call tools until the task is complete.
  • It then stops and returns the final reply.
  • Includes an end-loop guardrail to stop tool calling once outcomes are done (or when the system can’t proceed).

System prompt customization (“soul.md”)

  • The demo shows a local file soul.md in the Hermes folder acting as the system prompt.
  • Example personalization: the agent’s replies are forced to start with “pika pika…” based on sentiment/excitement/sadness.

Supported tools & capabilities (examples shown)

Local tools available to the agent (examples)

  • Terminal
  • Browser (web navigation / scraping-like actions)
  • Delegate task (spawn sub-agents)
  • Cron job scheduling (automated repeated actions)
  • MCPs mentioned as connectors/tools depending on the task

Demonstrated tool workflows

  • OS check & shell history via the terminal tool.
  • Cron job: “Pokémon/developer jokes every minute for the next 10 minutes,” scheduled via Hermes.
    • Speaker notes a limitation: WhatsApp delivery may not match expectations (example: only 2 jokes arrived).
  • Browser automation: Check a YouTube channel and create new video titles.
    • Handles a wrong URL by updating and retrying.
  • Python automation: Browser search + running scripts (via terminal) to fetch/transform content.

Memory system (local, multi-part)

Hermes splits persistent knowledge into local components:

  • Procedure memory (skills.md)

    • Stores “how to act” / operational behavior.
    • Saved as skills definitions (e.g., how to delegate to Claude CLI or how to use coding tools).

  • Semantic memory (memory.md)

    • Stores durable user facts (e.g., “favorite testing framework is pytest,” YouTube scraping quirks).
    • Speaker claims Hermes uses plain text + top-K keyword matching, rather than embeddings/vector similarity.

  • Episodic memory (state.db)

    • Stores chat history / dated events.
    • Over time, it’s consolidated (distilled) into semantic memory using auxiliary cheaper models.

Explicit vs implicit updates

  • Explicit memory saving: e.g., “Save to memory my favorite testing framework is pytest”.
  • Automatic updates after errors: e.g., saving a note that a previous YouTube URL was wrong and updating behavior accordingly.

Skills (procedural knowledge) management

  • Hermes has a skill management tool enabling:
    • explicit creation of skills (e.g., video script formatting style)
    • querying the created skill file
  • Speaker tests creating skills like video prep with specific stylistic constraints.

Parallelism & sub-agents

  • Delegate task can spawn multiple sub-agents in parallel.
    • Example: one sub-agent researching LM eval harness
    • Another researching VLM architecture

Sub-agent workflow example

  • Spawn a sub-agent that uses Claude CLI (headless mode) to generate and run a Python script:
    • fetch top 5 Hacker News stories
    • convert to Markdown
    • verify results

LLM Ops / evaluation support (limitations noted)

  • Speaker claims Hermes (as of their research) does not include built-in LLM ops/eval comparable to other setups.
  • What’s mentioned:
    • trajectory export / logs (trace tool calls and errors)
  • What’s not mentioned:
    • an evaluation framework (speaker contrasts this with LangSmith/Langfuse-style evaluation workflows).
  • Suggestion implied: eval may need to be built manually, since the system is local and custom.

Claimed differentiator vs similar agent harnesses

  • Compared to Clawcode/OpenClaw, Hermes is presented as more special because it:
    • auto-updates memory/skills locally as you use it
    • potentially offers privacy benefits (no cloud storage of knowledge)
  • Otherwise, core “harness + loop” behavior is described as standard for agent implementations.

Main speakers / sources

  • Primary speaker: The YouTube presenter (not named in the subtitles).
  • Project / source referenced: Hermes agent harness from Neural Research
    • repo/site: hermes-agent.newresearch.com

Original video