This list is what the client then uses for all subsequent connections to produce or consume data.The broker returns metadata, which includes the host and port on which all the brokers in the cluster can be reached.The client initiates a connection to the bootstrap server(s), which is one (or more) of the brokers on the cluster.On one is our client, and on the other is our Kafka cluster’s single broker (forget for a moment that Kafka clusters usually have a minimum of three brokers). It’s simplified for clarity, at the expense of good coding and functionality 🙂 An illustrated example of a Kafka client connecting to a Broker It’s very simple and just serves to illustrate the connection process. It’s written using Python with librdkafka ( confluent_kafka), but the principle applies to clients across all languages. It’s a fully managed Apache Kafka service in the cloud, with not an advertised.listeners configuration for you to worry about in sight!īelow, I use a client connecting to Kafka in various permutations of deployment topology. If the nuts and bolts of the protocol are the last thing you’re interested in and you just want to write applications with Kafka you should check out Confluent Cloud. To read more about the protocol, see the docs, as well as this previous article that I wrote. The broker details returned in step 1 are defined by the advertised.listeners setting of the broker(s) and must be resolvable and accessible from the client machine. What sometimes happens is that people focus on only step 1 above, and get caught out by step 2. If the broker has not been configured correctly, the connections will fail. The client then connects to one (or more) of the brokers returned in the first step as required.This returns metadata to the client, including a list of all the brokers in the cluster and their connection endpoints. The initial connection to a broker (the bootstrap).

When a client wants to send or receive a message from Apache Kafka ®, there are two types of connection that must succeed:

Cloudera docker on mac