Connecting a producer to a kafka cluster happens in two steps.
The first step is the bootstrapping process. This is the kafka brokers you initially configure in the kafka output in filebeat (or any kafka producer). On bootstrap, one of the configured brokers is choosen by random for the cluster meta data. The meta data do contain information about all known brokers, topics and partitions. The brokers
information do contain the brokers advertised addresses (hostnames and ports).
Step two is the kafka client spinning up a worker for each broker, each trying to connect to one of the advertised addresses.
When a kafka broker joins a kafka cluster, it broadcasts its public "advertised address" to all other brokers in the cluster, so they can update the cluster metadata. A common error is to have a broker advertise a wrong hostname (e.g. locahost) and or port. Check your kafka server config files for:
advertised.host.name
, advertised.listeners
, advertised.port
, host.name
, listeners
, port