How to Distribute Keys Securely for 20% of the Web

Key points (technological concepts & product/architecture features)

1) Problem statement: key leaks/mismanagement drive major breaches

• Recurring causes of key compromise:
  • Human error
  • Software vulnerabilities
  • Misconfiguration
  • Malicious attacks
• Consequence:
  • If an attacker obtains a private key/seed, they can decrypt sensitive data, access it, or enable malicious artifacts.
• Even if companies “have KMS,” issues can still occur due to:
  • Process/framework shortcomings
  • Lack of reliability and scalability

2) Critique of existing KMS frameworks (e.g., NIST guidance gaps)

• NIST 800-57-style guidance covers:
  • Lifecycle management
  • Best practices
  • Application-specific implementation patterns
• The speaker’s argument: missing resilience and scalability, specifically:
  • How to recover if a primary system fails
  • What happens if an entire cluster/region goes down
• Conventional cloud/region-pinned KMS designs treat outages as catastrophic because services depend on a region.

3) Evidence of KMS outages impacting real services

The talk references multiple cloud provider KMS-related outages:

• AWS (Nov 2020): region outage affecting KMS-dependent services (EC2, RDS, Lambda)
• Google Cloud (Sep 2020): connectivity outage severing an APAC region and affecting KMS users
• Azure Key Vault outage (Mar 202*): VM applications hanging; required reboot

It also mentions:

• Cloudflare keyless SSL outage
  • Demonstrates the risk is broad across providers.

4) Cloudflare’s goal: “distributed key management” for availability + scale

Cloudflare’s edge network characteristics:

• Extremely distributed footprint (edge in many cities/data centers)
• High performance requirement:
  • ~50ms target to 95% of the world
• Services run essentially on every server (not pinned to one region)

Why region-per-KMS replication models don’t fit:

• Failure cascades across services if one region is down.

Requirements for their system:

• Resilient/always available even when a KMS component is down or under maintenance
• Horizontally scalable across core and edge
• Performant (meets/exceeds edge constraints)
• Secure end-to-end, including secure key distribution/loading
• Open-source orientation to avoid vendor lock-in

5) Architecture: core KMS + replicated edge KMS (distributed/replicated key tier)

High-level “10,000 foot view” model:

Core KMS (central tier)

• Custom-built hardened Linux system
• Holds:
  • Managed key lifecycle
  • Access/admin functions
  • Policy database
• Creates managed keys and policies
• Periodically replicates managed keys/policies to edge KMS servers

Edge KMS tier (E-KMS)

• Stores replicated key rings and policies
• Can query core if a key isn’t present
• Synchronizes with other edge KMS nodes to maintain availability if core is slow/offline

KMS client (proxy/agent)

• Deployed locally on servers
• Forwards key-service requests from applications/services to intermediate/edge KMS servers
• Provides local proxying for resilience:
  • If one edge is slow/unavailable, requests can be forwarded onward

Key management detail:

• Managed keys have fine-grained identity- and policy-based controls (e.g., protocol identity, IP address)
• Hardware backing:
  • Core uses FIPS 140-2 Level 3 HSMs
  • Edge uses synchronized copies in FIPS Level 3 HSMs

6) Key-service request workflows (resiliency behavior)

• Workflow 1: normal operation

  • Application requests key-service via KMS client
  • If the data key is missing in cache, the request goes to edge KMS
  • Edge KMS processes request and updates client cache

• Workflow 2: edge unavailable/slow

  • KMS client forwards to another edge KMS or to core KMS
  • Core/other node processes it, updates cache, and responds

7) Local secure key loading: preventing “heartbleed”-style memory leaks

The second half focuses on the host/application side: once a key arrives, how to keep it safe.

Threat model:

• Keys live in memory address spaces, which are “flat” in typical OS/process models.
• If application code or libraries are compromised, keys can be accessed/exfiltrated.
• Example discussed: Heartbleed-style out-of-bounds read due to missing input validation, exposing an adjacent key buffer in memory.

Mitigations discussed:

• Agent model

  • Move cryptographic keys into a separate process
  • App calls the agent via a well-defined interface
  • Security benefit: isolation
  • Downsides: complexity (two programs), IPC, authentication, reduced flexibility

• Linux-kernel approach

  • Linux Kernel Key Retention Service / kernel keystore
  • Keys stored as kernel objects, outside the application’s address space.
  • Provides system-call interfaces and supports security policy hooks and lifecycle controls (including auto-destruct on process termination).
  • Enables key hierarchy via key rings (grouping concepts similar to KMS key rings/policy grouping)

8) Kernel keystore key types for avoiding plaintext exposure to user space

Two provisioning approaches:

• Encrypted keys

  • Applications provide an encrypted wrapped key
  • Kernel unwraps transparently using wrapping keys in kernel space
  • If other processes try to read it, they only see wrapped/encrypted form

• Trusted keys

  • Wrapping key derived via a TPM (if present)
  • Kernel unwraps using TPM operations
  • TPM operations can be slow, so a combined approach is described

Provisioning system described:

• TPM stores device management keys used to wrap/unwrap kernel keystore encryption and cache encryption keys
• KMS provides per-application wrapped keys; kernel unwraps them
• Goal: plaintext cryptographic material is never exposed to user space apps

9) Improving kernel key provisioning UX + security: request_key + helper modules

Problem with the basic approach:

• Applications still need to contact KMS to fetch wrapped keys
• Multiple KMSs complicate application provisioning
• Limited admin control can lead to accidental permission issues

Mitigation:

• Use kernel system call request_key instead of add_key
  • App provides a key identifier
  • Kernel requests payload on behalf of the process
  • Kernel invokes a user-space helper (commonly /sbin/request-key)
    • Helper can run a KMS client to fetch the correct key from the remote backend

Benefit:

• Decouples app code from KMS connection strings/protocols
• Centralizes key retrieval via an OS trusted interface

10) Putting it together: end-to-end workflow + cryptographic operation placement

• Boot/phased environment concept:
  • TPM acts as a trusted anchor
  • Trust extends into Linux kernel keystore
• Data keys:
  • Cached in kernel memory
  • Protected by cache encryption keys
  • Keystore protected by its own encryption keys

Two workflow categories:

• Workflow 1

  • Unwrapping happens in kernel
  • Cryptographic operation done by the app
    • (unwrap local, operations remote)

• Workflow 2

  • Cryptographic operations (decrypt/sign/generate) performed inside the keystore
  • Unwrapping handled remotely or separately as described in the steps

11) Conclusions: “distributed KMS” + “kernel key objects” to reduce operational and memory-leak risks

The speaker’s argument:

• Key leaks can still occur via mismanagement/framework failures—especially when KMS components/regions fail.

Proposed solution:

1. Fully distributed KMS with replication/consensus so keys remain serviceable under outages
2. Treat keys as kernel objects to reduce exposure to memory-safety vulnerabilities that could leak keys

12) Open documentation / resources

• The talk claims related work is blogged in stages.
• The speaker’s website is referenced for security research and updates.

---

Q&A highlights (from subtitles)

• Core KMS vs edge KMS

  • Core KMS:
    • Custom-built
    • Uses managed-key replication
    • Consensus handled only within the core tier (where keys are created)
    • Then replicated out as replicas to E-KMS

• Geolocation/geofencing question

  • Keys are bound to identity/policy
  • A custom protocol/service is used (details teased as future/Black Hat)
  • Network/policy tags label where a key should live

---

Main speakers / sources

• Speaker: Derek Chamorro (Cloudflare)
• Referenced source: NIST 800-57 (key management guidelines)
• Referenced example/attack: Heartbleed (memory disclosure)
• Referenced security concept: NSA article “Software or Memory Safety” (memory safety discussion)
• Referenced OS features/documentation:
  • Linux kernel key retention service / kernel keystore
  • System calls like request_key / add_key