Kafka Ingestion Example

This page provides an overview of how to create a simple Java Kafka producer that publishes data into a Kafka broker for consumption by the Stream Processor.

Kafka organizes streaming data into topics which are subdivided by numeric independent partitions.

Note

It is recommended for production deployments that Kafka be configured with TLS encryption. For a guide on setting up TLS with Kafka and the Stream Processor, refer to the Kafka with TLS guide.

This example is broken down into the following sections for setting up, running, and adding TLS.

For information on setting up a simple Kafka broker cluster, see the setup guide.
To see a pipeline configured to ingest from, see the ingestion example.
To learn more about creating and configuring TLS certificates, see the TLS setup guide.
To enable TLS ingestion, see ingesting with TLS.
To fit and apply a clustering model on a Kafka stream, see ML Clustering example.