Video summary

Вебинар о Managed Service for Apache Kafka®

Main summary

Key takeaways

Technology

Managed Service for Apache Kafka® in Yandex Cloud — Key Concepts & Features

What Kafka is (as described)

  • A system for streaming data processing and transmission using the publisher–subscriber (producer–consumer) pattern.
  • Designed for very high throughput, durable disk storage, and fault-tolerant operation.
  • Supports online consumption as new data arrives.
  • Scales horizontally by adding brokers to increase throughput.

Kafka vs. “message queues” (comparison)

The speaker contrasts Kafka with queue systems (example mentioned: RabbitMQ), noting Kafka’s built-in focus differs:

  • Kafka doesn’t provide out-of-the-box support for complex workflows such as:
    • advanced processing patterns,
    • request–reply messaging,
    • message priorities,
    • custom error routing/handling beyond core mechanics.
  • Such logic must often be implemented in application/business logic for Kafka use.

Kafka Core Model (entities and behavior)

Cluster

  • A set of servers (brokers) that store data and serve client requests.

Topic

  • Like a “table” conceptually: a named message space with its own settings, access rights, quotas, etc.

Partitions / “topic sections”

  • A scalable unit: each partition is stored separately and replicated independently.
  • Ordering is guaranteed within a partition.

Messages & offsets

  • Each message in a partition gets an immutable, monotonically increasing offset.
  • The producer receives the offset when writing.
  • The consumer reads from a chosen offset (either managed by the consumer or committed by Kafka in internal topics).

Keying

  • Producers can specify a message key so that messages with the same key go to the same partition (via hashing).

Replication and fault tolerance

  • Replication factor controls how many copies of partition data exist.
  • Leaders and replicas:
    • One broker acts as the leader for read/write for each partition; replicas store copies.
    • If a broker fails, leadership switches via controller coordination (speaker mentions ZooKeeper/keeper concepts).
    • When the broker returns, leaders may switch back to spread load more evenly.

Scaling behavior

  • Topics/partitions are distributed across brokers to enable near-linear performance scaling when adding brokers.

When Kafka should be used (scenario guidance)

Kafka is highlighted as a strong fit for:

  • High-volume streaming where throughput matters most.
  • Event and log aggregation
    • Example: clickstream events (user transitions, purchases, ad clicks).
    • Kafka can preserve ordering (with partition keys) and enable offline/secondary processing.
  • Log ingestion
    • Mention of tooling/plugins to store logs into Kafka for later processing by other systems.
  • Fan-out to many consumers
    • One event source feeding multiple downstream systems (mailer, analytics, other storage).
  • Event sourcing / microservices architectures
    • Kafka as a source of truth for services (fault tolerant), where microservices update local DBs.

Kafka is considered useful as:

  • Intermediate storage / buffering
    • e.g., bridging data between systems or supporting ingestion steps in architectures like “Lambda”.

Kafka may be harder when:

  • Very complex messaging/workflow logic is required
    • Kafka provides simpler producer/consumer semantics; advanced orchestration needs custom implementation.
  • Fine-grained per-message acknowledgment/handling
    • Kafka’s offset model typically requires implementing your own tracking if you need “ack one message, discard it, etc.”

Managed Kafka in Yandex Cloud — Control plane vs data plane

Goal

Provide a user-friendly deployment so users can provision Kafka “in a few clicks” without handling:

  • scaling,
  • updates,
  • failover cluster configuration,
  • cluster monitoring,
  • many replication/backup complexities (with defaults provided).

Architecture

Control Plane

  • Manages metadata (databases/users/settings).
  • Handles UI/CLI/API requests.
  • Monitors operations; manages backups and security controls.

Data Plane

  • Runs on isolated VMs in user-specific subnets.
  • Contains Kafka engine plus supporting tooling:
    • centralized log delivery,
    • update mechanisms,
    • backup creation for PITR-capable databases (Kafka itself discussed separately below).

Operation workflow (cluster creation)

  1. User request → internal API validates/authenticates.
  2. Target state written to a metadata DB (“metabase”).
  3. Workers execute long-running async tasks:
    • create VMs,
    • set up TLS, DNS, cluster entities,
    • apply settings (passwords, databases),
    • finalize operation state.

Deployment options & resiliency considerations

  1. Test cluster

    • Single broker in one zone/availability zone.
    • Uses one keeper host; no meaningful replication, so maintenance/broker downtime affects access.
    • Recommended only for testing/hypotheses—not production.

  2. Production (“combat”) cluster

    • Requires at least 2 brokers, ideally 3 brokers across zones.
    • Supports resilience: if one broker is shut down, read/write access remains.
    • Uses separate keeper hosts across availability zones for controller coordination.

Reliability & write acknowledgment settings

Producer write acknowledgement behavior

  • “Fire and Forget”: don’t wait for broker confirmation.
  • Intermediate mode: wait for confirmation from leader only, not all replicas.

Topic-level and cluster-level replication acknowledgment logic

  • Min In-Sync Replicas (minins, also described with “syn” wording)
  • Recommended defaults:
    • Replication factor = 3
    • Min in-sync replicas = 2
  • Allows availability even if one broker is lost.

Topic overrides

  • Topic can override cluster defaults:
    • set minins = 1 for a particular topic to prioritize availability over strict write durability/reading reliability tradeoffs.

Practical UI/API features and user workflow (tutorial-style)

  • Cluster creation in UI

    • Configure:
      • Kafka version,
      • resources (CPU/memory/storage),
      • availability zones,
      • network,
      • number of brokers per zone.
    • Start cluster creation operation; monitor operation status.

  • Topic creation

    • Set:
      • replication factor,
      • number of partitions,
      • retention/cleanup policy,
      • topic settings (defaults used if omitted).

  • User creation & topic permissions

    • Create users and assign access rights for producing/consuming topics.

  • Connection settings

    • UI shows connection details for console utilities and languages such as Python, Java, Go.

Security, networking, and operational choices

Encryption

  • Traffic between brokers is encrypted; Kafka uses SSL interface.
  • If “public access” is enabled, non-SSL (public) access is disabled, forcing SSL connectivity.

No direct keeper access

  • Users manage Kafka cluster via UI/API; keeper is not exposed to users.

Backups

  • The speaker states Kafka backups are not provided like traditional database backups (Kafka’s model makes them considered unnecessary).

Storage roadmap

  • Mentions planned “Layered Storage”:
    • potentially move older messages to S3 when feature is ready (implementation-dependent).

Scaling roadmap

  • Already added/available:
    • support for white IP addresses (working).
  • Upcoming:
    • vertical scaling (increase memory/disk/CPU without losing availability, if configuration supports it),
    • horizontal scaling (more brokers / more availability zones),
    • log delivery enhancements.

Practical example tutorial: clickstream → Kafka → ClickHouse → Data Lens (and visualization)

Use case

  • Backends generate events (example: clickstream/user transitions).
  • Store securely in Managed Kafka.
  • Enrich events with reference data from PostgreSQL.
  • Query in ClickHouse.
  • Visualize with Data Lens integrated with ClickHouse.

Data flow described

  1. Generate artificial clickstream-like events.
  2. Load them into Kafka via managed Kafka.
  3. In ClickHouse:
    • create a table using the Kafka engine pointing to Kafka connection parameters.
    • nuance: the table receives only the latest messages; to handle “reading from offsets reliably,” a materialized view is used to persist data into ClickHouse.
  4. Add a reference dictionary in PostgreSQL:
    • register external dictionary settings in ClickHouse (foreign-key-like access using user IDs).
  5. Query in ClickHouse using event stream + reference dictionary.
  6. Visualize in Data Lens:
    • connect to ClickHouse,
    • create charts (example shown: pie chart / distribution by geo zones).

Resources

  • Source code referenced as available in an open GitHub repository.
  • Encourages users to copy/extend it; missing pieces can be added via repository contributions.

Speaker(s) / sources

  • Vyacheslav Ksenz — main presenter/speaker (Yandex Cloud), explaining Kafka managed service architecture, features, and the practical example.

Original video