Building a Kubernetes-native Experience for Your Multi-cluster Fl... Andrea Tosatto & James Munnelly

Goal / Big idea

• Present ongoing research into extending Kubernetes with a thin multicluster API server layer so users get a Kubernetes-native multicluster experience.
• The approach abstracts placement, aggregation, and consistent reads across multiple clusters without forcing users to manage cluster-specific details.

Motivations and platform pain points (scalability + operability)

• API server scalability limits have long impacted platform teams once they expose the Kubernetes API to many clients.

Common mitigations mentioned:

- **Separate high-churn events** into another etcd/cluster.
- Use **binary/protobuf encoding** to reduce memory usage.
- Ensure **strong CRD validation** to avoid serialization/memory issues.

• Community improvements called out:

  • Cyber/serialization work, notably around handling JSON-encoded CRD objects and list responses.
  • Consistent reads from the cache, to avoid incorrect load patterns hitting etcd (e.g., misuse of resourceVersion semantics).
  • Fixes around server-side filtering (avoiding filtering after retrieving all objects from etcd).

• Beyond scaling API servers alone:

  • Scaling controllers: more objects → more controller work.
  • Operational risk: larger platforms increase blast radius of configuration/outage events.
  • Need for platform abstractions that provide:
    • Well-defined domain-specific workloads
    • Topology information (e.g., spread across maintenance / “four domains”)
    • Improved utilization without users constantly tuning requests/limits.

Architecture: “Cluster as a Service” but beyond one cluster

• Traditional model: each team gets its own cluster.
• New model: allow customers to scale beyond a single-cluster boundary while keeping infrastructure details hidden.

Key building blocks:

1. Multicluster scheduling (workload placement layer)
2. Multicluster API aggregation (API server layer)
3. A consistent semantics layer for list/watch, plus proxying for patch/update/delete

Multicluster scheduling using MultiQ

Instead of submitting workloads to a single cluster, users submit to a multicluster API server.

MultiQ receives the workload and:

• Reconciles it into an internal “deployment-like” representation.
• Finds an appropriate cluster that can run it.
• Pushes down spec to the target cluster.
• Syncs up status back to the multicluster API view.

Design intent

• Similar abstraction to Kubernetes Deployments/StatefulSets/Jobs.
• Use MultiQ not just for batch, but as a general multicluster workload framework.

Collaboration / extensibility

• Collaboration with upstream around elastic workloads and supporting flexible scaling semantics for Deployments/jobs.
• MultiQ supports custom/“external” controllers to orchestrate behavior beyond core Kubernetes workloads.

How the scheduler chooses clusters

• MultiQ provides placement behavior (e.g., sequentially trying clusters).
• It can be extended with custom placement logic (e.g., avoid a cluster marked for maintenance via taints).

Reconciliation model

• Spec is pushed down; status is aggregated up, so users perceive a single consistent view.

Multicluster API server aggregation (core contribution)

Requirement: preserve Kubernetes “list/watch” semantics

• Controllers should be able to talk to the multicluster API like they talk to a normal Kubernetes API.
• Main challenge: resourceVersion semantics
  • resourceVersion is a point-in-time reference from etcd; controllers use it to establish watches.
  • In multicluster, there are multiple etcd instances—so you need a monotonic version across clusters.

Approaches considered (and why they fail)

1. Vector resourceVersion (per-cluster resourceVersion list)

   • Hard because many clients assume resourceVersion is a parseable integer/format and may rely on it.
   • Breaks ordering complexity when clusters are added/removed.

2. Sync all objects to the multicluster layer

   • Works but creates scalability problems (e.g., a “gigantic etcd” storing all objects).
   • Violates the “thin layer” reliability/scalability goals.

Final approach: “reclocking events” with a Version Marker object

• Instead of syncing all objects, the system focuses on producing a monotonically increasing version number across clusters.
• Introduces a virtual marker (referred to as “version marker object”) stored in etcd to track event ordering/visibility.

Version marker metadata (high-level)

• store: which underlying cluster the event originates from
• object: reference to originating object identity and originating cluster resourceVersion
• template: includes the generated multicluster-wide resource version (the monotonic ordering token)
• Optimization mention:
  • Supports label selector logic at the multicluster layer.

Two marker types:

• Bookmarks: maintain continuous tracking so watches can resume efficiently even during idle periods.
• Mapping markers per aggregated resource (e.g., per Pod) indicating which cluster/version produced it.

How clients experience the API

• Client issues get/list/watch to the multicluster API server.
• The multicluster server implements the API server storage interface using:
  • a “virtual marker store” (instead of storing all real objects),
  • a request routing layer that translates multicluster resourceVersion back into the correct underlying cluster’s resourceVersion.

Other operations: patch/update/delete

• For non-read operations, the multicluster API server proxies requests:
  • finds which cluster actually hosts the object (via marker mapping),
  • rewrites the multicluster resourceVersion to the underlying cluster’s real one,
  • forwards the request.

Logs and exec

• Supports logs and exec using the same mapping/proxy methodology.
• Demo emphasizes that from the user’s perspective it feels like standard Kubernetes.

Demo highlights (what was shown)

• Two local Kubernetes clusters running with:
  • multicluster API server
  • etcd
  • MultiQ

Showed:

• An API aggregation configuration defining which resources and clusters are aggregated.
• Virtual marker objects demonstrating:
  • which cluster a Pod exists in,
  • multicluster resourceVersion mapping,
  • underlying cluster resourceVersion.
• Creating a Deployment and Job and observing:
  • Pods being scheduled across clusters (via MultiQ).
  • watch pods reflecting state transitions and Job completion through multicluster streaming.
• logs and exec working through the multicluster API.

Q&A / clarifications (key technical points)

• Support for custom resources / CRDs

  • Yes: aggregation is driven by an API aggregation object that specifies:
    • resource kind/scope,
    • which underlying cluster(s) to aggregate from.
  • Works for core types and CRD types (and can support user-defined abstractions).

• Where aggregation data is stored

  • Virtual markers can be stored similarly to events, potentially distributed/sharded across etcd instances; marker storage is separate from storing full objects.

• How MultiQ reconciliation propagates

  • MultiQ pushes spec down and syncs status up.

• UI/clients (dashboard, kubectl)

  • Can run the Kubernetes dashboard against the multicluster API server; kubectl can use a single kubeconfig to interact as if it were one cluster.

• Component deployment model

  • MultiQ consists of:
    • Q controller-manager (“Q” piece) and related control components
    • Two copies of Q manager/controller:
      • one configured to talk to the multicluster API server,
      • one running on each underlying cluster to evaluate/admit workload based on local capacity.

Main speakers / sources

• Andrea Tosatto (primary speaker)
• James Munnelly (co-speaker referenced; not present for part of the talk)

Referenced project/community components

• MultiQ (mentioned as a community-generated project)
• Kubernetes SIG multicluster (driving multicluster work upstream)
• etcd (CD layer used for resourceVersion semantics)