Андрей Воронков и Руслан Махмудов. Chained Together: безопасность композиций агентских скилов

Main ideas / lessons

• Individual skill security isn’t enough: Even if each agent “skill” (tool/action module) is safe when tested alone, combining multiple skills in the same agent workflow can create new, dangerous capabilities.
• Skills are small packages with executable behavior + instruction text: A skill is essentially a folder containing a main descriptor (e.g., skill.md) plus resources like executables/scripts/assets.
• Automation is necessary for testing: Humans often don’t read full skill contents before using them; skills can also contain complex or hard-to-analyze code.
• Security must cover both:
  1. Skill safety (behavior vs. declared description, tool/resource usage, vulnerabilities, policy compliance).
  2. Skill composition safety (the risk created when skills operate together).

---

Methodology: Skill security approach (Yandex “Skill Store” + automated checks)

1) Publish/ingest skills through a controlled registry

• Yandex implemented Skill Store—a registry interface on top of their internal Arcadia platform (described as an analogue of GitHub).
• Developers publish skills via pull requests.
• Security review runs on PR submission to issue comments/verdicts and prevent unsafe skills from entering infrastructure.

2) What to test in each skill

Description / behavior alignment

• Check that the skill’s actual behavior matches skill.md.
• Detect cases where a skill does something other than what it claims (e.g., potential “injection” via malicious or misleading instruction text, or any undeclared functionality).

Tools & external resources

• Identify what endpoints/tools/resources the skill can access or use.
• Use this to proactively block skills that use disallowed tools/resources.

Code & dependency security

• Run static vulnerability checks and dependency scanning (described as classic SAST/DAST-like checks; also mentions an “lm … SAS check” in the general sense).
• Prevent vulnerabilities from being admitted to the registry.

Policy compliance

• Verify that the skill follows company security rules (the agent is treated as a full participant in infrastructure).

3) Verification architecture (parallel checks + final decision)

• Skills arrive via PR, then multiple analyzers run in parallel.

• There are 3 main analyzers (“cubes”):

  1. Description/Behavior validator (called Description Mater Security Scanners in the transcript)
     • Main idea: check actual behavior matches declared skill.md, including detecting obscurity/undeclared behavior.
  2. Endpoint/Tool Extractor
     • Extracts the set of tools/endpoints the skill can use.
     • Helps enforce access control granularity; blocking can depend on these exposed endpoints.
  3. LM-focused security scanner (mentions “LM SAS”)
     • Collects a list of vulnerabilities found, with special attention to cases involving conditioning and local access risks.
     • The transcript emphasizes preventing dangerous outcomes like unintended injection leading to local compromise.

• Results go to a final component called Security Arbiter, which enforces policies and produces verdicts.

4) Verdict outcomes (after Security Arbiter)

• Pass: if no violations → allow the skill.
• Warning: if complaints exist → issue warning, and depending on criticality, either:
  • allow with required fixes, or
  • force developer adjustments.
• Fail (blocked): if it doesn’t meet security requirements → never allow usage.

5) Runtime/policy enforcement examples mentioned

• MCP requirement for critical systems
  • Critical infrastructure requires MCP, with an MCP Gateway that checks external models and use of certain tools.
  • A goal is a common AI gateway so skills can bypass MCP while still being under security controls.
• Blocking remote control / risky access
  • Remote control is described as too dangerous in company infrastructure (e.g., losing a phone, downloading an APK).
  • Skills enabling such remote control are blocked.
• Secrets/sensitive data handling
  • If sensitive data or secrets are found during verification → block and require developer remediation.

6) Example: pull request–based checking workflow

• Similar to GitHub PRs:
  • detect which skills changed in a PR,
  • run analyzers for each impacted skill,
  • produce verdicts.
• Example given: a skill claimed “readonly” but contained only a single file—used to illustrate how Security Arbiter flags issues based on analyzer results.

7) Performance/cost constraints (as stated)

• Approx cost: ~$10 per day per internal inference on GLM 52 (wording in transcript is noisy, but the cost scale is given).
• Around 450 checks/day.
• A single check can hit ~2 skills on average.
• Testing time: up to ~2 minutes per check (limited by the longest-running analyzer).
• Desired improvements:
  • more deterministic checks,
  • use smaller/cheaper models for simple tasks rather than always GLM 5.2,
  • reduce hallucinations/false positives,
  • provide a local checker for developers before PR submission.

---

Methodology: Skill composition risk (Ruslan’s part)

1) Definition of “skill composition”

• Skill composition = using two or more skills together by an agent.
• Risk arises because the combination enables behavior not apparent when testing each skill alone.

2) Simple example demonstrating emergent risk

• Skill A: reads local environment and transmits an ID.
• Skill B: changes network rules.
• Individually, each skill is “safe,” but together:
  • the agent obtains data (C runner/ID) via Skill A,
  • then uses Skill B with that ID,
  • enabling access from outside (undesired capability).

3) Why certain compositions are especially dangerous

• Most dangerous combos happen when “party” skills integrate with high-impact systems like:
  • GitHub, Warehouse, AVS, etc.
• Example described:
  • a message arrives (e.g., via Slack),
  • the agent treats it as instructions,
  • with GitHub skill access, it may produce a dangerous commit/merge, etc.
• The danger is framed less as “prompt injection gets in” and more as:
  • accidental/unintended actions enabled by high authority + multiple skills working together.

4) Composition risk analysis architecture (two main parts)

• Build a strict scheme from the skill’s code/tool calls:
  • represent what a skill can do in abstraction (e.g., read/write files, return values).
• Analyze the semantic scheme from the natural language skill instructions:
  • use an LLM to infer what capabilities the skill claims to enable (and what could happen “if the agent had all powers”).
• Scoring
  • evaluate danger levels (e.g., internet access → higher risk; reading secrets → high risk).
  • note: not all scoring is done via LLM because it can be unreliable; more deterministic checks are preferred for many big dangerous combinations.

5) Deterministic identification of risky combinations

• Evaluate chains/sets of skills:
  • determine what actions the combination enables,
  • mark combinations as dangerous or acceptable.
• Mitigation options suggested:
  • Combinational allow/deny list
    • e.g., allow A+B, allow A+C, but deny A+B+D because D increases risk.
  • Runtime policy enforcement (Poly Engine approach, described as a research prototype)
    • intercept agent actions and block dangerous ones at runtime.

6) “Don’t write your own policy engine” note

• Use existing frameworks for policy enforcement rather than building from scratch.

7) Benchmarking and testing

• Runtime enforcement was tested using an internal benchmark for false positives.
• Mention of a public benchmark: “Skill Composition Risk”
  • includes cases with combinations of skills (including some with 3 skills).
• Benchmark framing:
  • AB neutral: agent gets task instructions; agent acts and may unintentionally do bad things.
  • “Danger comes from indirect authority”: the combination can make success rates higher.
• High-level results:
  • Even newer/more capable models were not necessarily safer in composition:
    • using single skill sometimes yields low attack success (example: ~1%),
    • adding a second skill increases attack success (example: ~50%).
  • Poly Engine reduced attack rate down to 0%, but introduced false positives:
    • blocked some legitimate scenarios (~6% in one reported test).
  • Claim: model capability alone does not guarantee safety:
    • GLM 5.2 did not become safer; newer models can be more dangerous in composition.

8) Practical mitigations besides policy engines

Recommendations listed:

• Least privilege
  • run agents with minimal permissions: no internet / limited access → safer.
• Use sandboxes
  • sandboxing cuts risky scenarios substantially.
• Split workflow across agents
  • instead of one agent doing the whole chain, divide tasks:
    • one agent has limited skills for one action,
    • passes results to another with a different, non-overlapping set of skills.
• Guards / guardrails
  • “guards” help prevent many scenarios, particularly because composition risk is about unintended actions, not only prompt injection.

9) Key takeaway stated at the end of Ruslan’s section

• Check single skills for harmfulness, and check combinations for unintended actions—newer models may still be vulnerable to composition-driven failures.

---

Speakers / sources mentioned

Speakers (presenters in the video)

• Andrey Voronkov
• Ruslan Makhmudov
• Boris (asked a question; name appears as “Boris” in the transcript)
• Elizar (asked a question; addressed in the transcript)

Sources / referenced systems & documents

• Yandex internal systems
  • AI security team at Yandex (context)
  • Arcadia (internal platform)
  • Skill Store (Yandex registry interface)
  • Security Arbiter (final verdict system)
  • MCP Gateway / MCP (policy gateway and requirement)
• Benchmarks
  • “Skill Composition Risk” (public benchmark referenced)
• Tools / frameworks mentioned
  • SAST/DAST-style security checks (described generally)
  • Policy engine / Poly Engine (runtime interception approach)
  • Mentions “frameworks” for policy enforcement (not named specifically)