Transforming Data Streams with Kafka Connect: An Introduction to Single Message Transforms
0 likes1,149 views
The document introduces Kafka Connect and its Single Message Transforms (SMTs), which are functions to modify records during data streaming between sources and sinks. It covers common use cases such as data enrichment and partitioning, along with built-in SMTs and example configurations. Additionally, it discusses alternatives like Kafka Streams and ksqlDB.
Transforming Data Streams with Kafka Connect: An Introduction to Single Message Transforms
- 1. Transforming Data Streams with Kafka Connect: An Introduction to Single Message Transforms Ranjan Rao
- 2. ❏ Kafka Connect Overview ❏ Single Message Transforms ❏ Common Use Cases ❏ Built-in SMTs ❏ Example SMT Configuration ❏ Alternatives @ranjan_rao
- 3. Kafka Connect Overview ➔ Tool for streaming data from various sources to Kafka and from Kafka to various sink systems. @ranjan_rao Source Source Connector Sink Sink Connector
- 4. Single Message Transforms ➔ Provides a powerful and flexible way to transform data ➔ Simple function that accepts one record as an input and outputs a modified record ➔ Apache Kafka 0.10.2.0 @ranjan_rao
- 6. ✓ Kafka Connect Overview ✓ Single Message Transforms ❏ Common Use Cases ❏ Built-in SMTs ❏ Examples of SMT Configurations ❏ Alternatives @ranjan_rao
- 7. Common Use Cases ● Data enrichment/enhancement ● Partitioning ● Field masking ● Event routing @ranjan_rao
- 8. ✓ Kafka Connect Overview ✓ Single Message Transforms ✓ Common Use Cases ❏ Built-in SMTs ❏ Example of SMT Configurations ❏ Alternatives @ranjan_rao
- 9. Built-in SMTs ● InsertField ● ReplaceField ● MaskField ● ValueToKey ● ExtractField @ranjan_rao ● TimestampRouter ● RegexRouter ● InsertHeader ● HeadersFrom ● DropHeaders
- 10. ✓ Kafka Connect Overview ✓ Single Message Transforms ✓ Common Use Cases ✓ Built-in SMTs ❏ Example of SMT Configurations ❏ Alternatives @ranjan_rao
- 11. Example SMT configuration {“Firstname”: “John”, “LastName”: “Doe”,”Phone”: “000-000-0000” } {“Firstname”: “John”, “LastName”: “Doe”, “Phone”: “123-456-7890”} InsertField MaskField {“Firstname”: “John”, “LastName”: “Doe”,”Phone”: “ 000-000-0000”, “Origin”: “Source Connector”, “RecordTimestamp”: 1707940094 }
- 12. Example SMT configuration "transforms": "MaskField,InsertField", "transforms.MaskField.type": "org.apache.kafka.connect.transforms.MaskField$Value" , "transforms.MaskField.fields": "Phone", "transforms.MaskField.replacement" : "000-000-0000" "transforms.InsertField.type": "org.apache.kafka.connect.transforms.InsertField$Value", "transforms.InsertField.static.field": "Origin", "transforms.InsertField.static.value": "Source Connector", "transforms.InsertField.timestamp.field": "RecordTimestamp" {“Firstname”: “John”, “LastName”: “Doe”,”Phone”:“000-000-0000”, “Origin”: “Source Connector”, “RecordTimestamp”: 1707940094 }
- 13. ● Single Message Transforms for Confluent Platform ● Single Message Transforms for Kafka Connectors in Confluent Cloud @ranjan_rao Example SMT configuration
- 14. ✓ Kafka Connect Overview ✓ Single Message Transforms ✓ Common Use Cases ✓ Built-in SMTs ✓ Example of SMT Configurations ❏ Alternatives @ranjan_rao
- 15. Alternatives ➢ Kafka Streams ➢ ksqlDB ➢ Apache Flink @ranjan_rao
- 16. Thank You!