Connectors, Configuration, Converters, and Transforms | Kafka Connect 101 (2023)

Key Components

• Connectors:
  • Facilitate interaction between Kafka Connect and external technologies.
  • Responsible for defining where data should be copied to and from.
  • Implemented as connector plugins, which can be reused without coding.
  • Examples include:
    • Debezium MySQL Source Connector: Reads events from MySQL using the bin log.
    • Elasticsearch Sync Connector: Writes data from Kafka to Elasticsearch.
    • S3 Connector: Acts as both a source and sink for data to/from S3.
• Converters:
  • Handle serialization and deserialization of data between Kafka Connect and Kafka.
  • Important for maintaining schema information, which serves as a contract between pipeline components.
  • Common converters include Avro, Protobuf, and JSON.
  • Converters can be set globally or overridden per connector instance.
• Transforms:
  • Optional components that modify data as it flows through Kafka Connect.
  • Known as Single Message Transforms (SMTs), they can drop fields, add metadata, change data types, or rename fields.
  • More complex transformations may require a full stream processing layer like k-sqlDB or Kafka Streams.

Configuration and Usage

Connector instances are defined in configurations, specifying the necessary properties based on the technology. Kafka Connect abstracts much of the complexity, allowing users to focus on configuring connectors. The video also mentions using the Kafka Connect REST API and Confluent Cloud Console for managing connectors.

Practical Application

The video concludes with an invitation to apply the learned concepts in a hands-on exercise, emphasizing the ease of transforming input records before writing them to a Kafka topic.

Main Speakers/Sources

• Danica Fine