Video Summary - Hackbots | DEF CON 34

Main ideas / lessons

What a “hackbot” is today
- A hackbot is an offensive-security agent that uses LLMs and/or traditional ML to perform security tasks.
- People are building agents across bug bounty, red teaming, and offensive workflows—especially for web/dynamic testing.
Why hackbots are useful
- Automation + scaling: helps execute complex offensive workflows and scale to many sites beyond what a human can do alone.
- Repeatability: reduces the risk testers skip steps or hand off work inconsistently.
- Intuition assistance: AI can help develop tester intuition, though the talk focuses mostly on scaling/automation.
Key limitation: humans-in-the-loop are still necessary
- Even with agent frameworks, AI may:
  - Overestimate severity of findings.
  - Miss critical combinations (e.g., multi-step auth/redirect/OAuth chains needed for account takeover).
  - Miss obvious issues.
  - Behave unpredictably (example: agents performing DELETE actions despite “secure setup” and prompting).
- Therefore, the talk emphasizes mandatory “human in the outline” / specialist review rather than full autonomy.

Systems and frameworks discussed

Agent frameworks (tools people likely already know)
- Claude Code / Codex / Open Code (agent frameworks connecting LLMs to command line + tools).
- Maestro (agent management framework for running techniques by splitting work into smaller pieces).
- General architecture described
  - A thick client or terminal UI (TUI) that wraps multiple LLM instances.
  - Adds agents and tools such as:
    - Web data retrieval
    - Shell/terminal execution
    - File access
    - External API calls
    - Browser control
    - Code sandboxing
Skills (how the system is modularized)
- “Skills” are defined in Markdown and shared across frameworks via a common approach.
- A skill typically has:
  - A skill.md file with name + description (light context loaded initially).
  - Implementation/execution logic loaded only when the skill is invoked.
  - Optional arguments:
    - Model selection
    - Reasoning effort level
    - Context configuration (inherit prior conversation vs run independently)
    - Forking
  - Supporting assets:
    - Additional markdown files
    - Scripts in a scripts/ folder
    - Examples of expected output
“Completion bias” (why methodology must be broken into small steps)
- If you give agents long multi-step processes, they may do only the first portion.
- Mitigation:
  - Make methodology parts small
  - Keep skills short
  - Use separate tasks/agents for each step

Detailed methodology: building a high-quality hackbot skill (JS Analyze)

The talk’s “deep application testing” example is JS Analyze, a core Armada capability for JavaScript analysis to extract endpoints/APIs and understand client-side logic.

Goal / outputs

Extract all endpoints including:
- Current endpoints (from live site content)
- Historical endpoints (from archives like Wayback Machine)
Parse/map:
- Parameters
- JavaScript functions to API routes
- Organize by sections (e.g., admin vs user)
Perform technical analysis:
- Find secrets (including high-entropy strings when regex isn’t known)
- Locate dangerous client-side “gadgets” / functions
- Extract interesting comments and overlooked developer notes
- Identify tech/framework usage and historical attacks
Produce a report including:
- A summary report
- curl commands for each call (so other tools/agents/devs can reuse findings)

Step-by-step design approach (how it is engineered)

Start from a mind map of:
- What the tool should do (endpoints + history + parsing + reports)
- What inputs it needs (JS collection methods, archives, devtools)
- What output formats it must generate
Select best-suited tools for translation and analysis:
- Convert JavaScript to AST and map fragmented code:
  - Mentioned tools: JS Sleuth, and later “dominant” Jex Boss Scout
- Use archive tools: Wayback Machine and related snapshot services
- Use browser inspection for current endpoints and JS:
  - Chrome DevTools (as described)
Expand scope based on what is commonly missed:
- Embedded JS inside HTML (don’t only parse .js files)
- Lazy-loaded JS (loaded only after interaction/auth/click)
- Source map discovery
  - Search for .map files per JS asset
  - Look for directories/other domains where maps might live
  - Tip: mobile apps sometimes ship source maps for the same website JS
Deal with anti-automation systems:
- Do not rely on blocked fetch methods (the talk mentions avoiding WebFetch and sometimes even curl for scanning)
- Use Interceptor instead to avoid bot blocks (see Interceptor section below)
Increase analytical depth:
- Deobfuscation (turn minified/obfuscated code into readable form)
- Parse and derive:
  - API routes
  - AST structures
  - external symbols
- Extract actionable data:
  - parameters (including feature flags / admin-like functions)
  - framework tech signals
  - secrets + high-entropy strings
- Assess third-party library issues (vulnerabilities/outdated dependencies)

Split into micro-skills to avoid failure

Once the scope gets too large, split:
1. Search/loading skill (collect JS, archives, maps, etc.)
2. Parsing/deeper analysis skill
This supports completeness and reduces ignore/partial execution due to completion bias.

Verification strategy (completion + anti-hallucination)

Define expected outputs and structure using Markdown checklists.
Use a separate validation/review skill:
- Example described:
  - JS Analyze runs steps A–F and writes artifacts.
  - A separate verification skill checks:
    - checklist items are checked off
    - evidence artifacts exist
    - logs/artifacts have non-zero size
- If missing:
  - It nudges/retries the exact missed section
  - Uses general instructions to “find what you missed and complete it”
- Mentioned validator:
  - Uses Gemini for validation (“review skill”)

Tool-calling rigor (prevent syntax/format mistakes)

Don’t write prompts too loosely:
- If you say “scan with HTTPX,” the agent may choose an undesired syntax (e.g., scanning with the “wrong options”).
Always specify full tool syntax and explicit parameters (ports, headers, etc.).

Interceptor (browser control for dynamic analysis)

What it is
- A tool described as a browser extension that lets AI control browsing more effectively than common routing methods.
- Compared to Puppeteer/Playwright, but positioned as “10x better” (per speaker).
Why it’s used
- Avoid bot protections that block other approaches.
- Enable a “Dash Spider” style:
  - click every button
  - follow links
  - download/parse code
  - find JS loaded from other domains
Dynamic testing workflow described
- Intercept traffic locally and analyze:
  - requests made before redirects
  - JS loaded dynamically
- Helps with timing-sensitive scenarios (e.g., IDP race condition)

Context engineering / classifier-bypass tactics (and why they matter)

The speaker argues that building hackbots for offensive security involves dealing with model classifiers/refusals and “rejections,” so they use context engineering and authorized access pathways.

Trusted Access / CVP programs (Anthropic)
- Recommended:
  - Apply to programs for offensive use to reduce refusals.
  - Response times described as sometimes as fast as one day (speaker’s experience).
If program access isn’t possible: engineering around rejections
- Create a “scope of work” artifact (e.g., PDF) placed in the filesystem to support authorization claims.
- Context overloading / positive framing
  - Fill long conversations with language like “authorized security testing” and gradually lead into the request.
“Fable” / classifier workaround described
- A previously observed period where offensive tasks were blocked unless using a specific function path:
  - /BTW (function name as described)
- (Presented as historical experience; not positioned as permanent.)
Reported impact statistics
- Before these techniques:
  - success rates sometimes as low as 0.4%
  - missing data artifacts, empty directories, no tool logs, impossible-fast runs (e.g., completion in ~2 minutes)

Operating hackbots in practice: multi-model/multi-run ensemble

Main point
- To make hackbots reliably effective, run many models and many launches due to non-determinism.
Example launch described
- Multiple sessions across different model versions (speaker mentions Opus 4.6, Opus 4.8, ChatGPT 5.6, Kimi, GLM, and a custom model).
- Runs across many micro-skills (example: 120 micro-skills) and repeated multiple times per cycle.
Cost-performance principle
- Cheaper models can be used heavily; more powerful models better handle complex engineering.
Extra robustness
- Also include “bitter pill” approaches for large models (described conceptually below).
- Use an aggregator agent to merge outputs into final reports.

“Bitter pill” vs “true skill engineering”

“Bitter pill” testing
- Concept: instead of extensive step-by-step engineering, provide a high-level hint and let the model try.
- Works for some workflows, especially where simple guidance is sufficient.
- Speaker says it is not suitable for deep application-based testing.
True skill engineering
- Requires explicit domain knowledge codified into:
  - skills
  - structured steps/checklists
  - deterministic outputs/verification
- Central example: JS Analyze.

Practical best practices / engineering cautions

Don’t clutter skill directories
- Avoid writing outputs inside the skills folder; use separate project/output directories.
Always reference full paths
- Ensure the agent can’t “lose” tools/paths and that outputs are stored reliably.
Use safe mode when testing new models
- Prevent loading existing “me” identity/context or skills.
- Remove redundant prompts over time if the model performs well on its own.
Be careful with workflow automation
- Workflow frameworks can terminate or block tool calls via classifiers.
- Errors may be subtle/red in GUI or in panels; users may not notice missing results.

Examples / case stories (what the hackbots found)

Admin password reset / parameter brute forcing
- Hackbot found an unusual “forgot password” page.
- Human noticed two recovery pages and guided the agent.
- Agent brute-forced likely ID parameters (e.g., “admin”) and set a default password.
- Lesson: create separate skills for specific edge-case flows.
IDP race condition
- Briefly visible single-page app content before redirect to SSO provider.
- Agent created a skill using Interceptor to pause/block redirects and harvest pre-redirect API requests.
- Reports generated curl commands; used without authentication.
- Lesson: build “research skills” for discovery of these timing vulnerabilities.
SSRF leading to Kubernetes/Envoy config dump
- Initial SSRF agent biased toward outdated AWS metadata targets.
- It expanded to localhost ports and found Envoy admin on port 9901.
- From there, a config dump endpoint exposed tenant API keys for many users.
- Lesson:
  - Break SSRF into phases:
    - identification
    - post-exploitation with locally hosted admin panel focus
  - Training-data methods may be outdated; codify updated expertise in skills.

Speakers / sources featured

Speakers

Jason (main speaker; introduced as “My name is Jason.”)

Mentioned people/framework names/tools (sources)

Johnny 5 (referenced humorously as a “friend,” likely representing an example/character; not substantiated as a real collaborator in the subtitles)
Andriy Karpathy (mentioned as inspiration for long “AI jam” sessions)
Anthropic (Trusted Access / CVP programs; models and “Fable” classifier described)
Gemini (used for the verification/review skill)
Claude Code / Codex / Open Code
Maestro
XMind (mind-mapping tool)
Wayback Machine
JS Sleuth
Jex Boss Scout (as described later/dominant)
ffuf (content discovery fuzzing)
Interceptor (browser interception extension/tool)
Chrome DevTools
HTTPX (subdomain scanning tool)
Puppeteer / Playwright (comparators for Interceptor)
OneLogin / Octa (SSO providers referenced)
Burp / Caido (mentioned as older routing tools they moved away from)

Hackbots | DEF CON 34 | BBV Masterclass

Key takeaways