The Beats Kafka Output plugin seems to be based on Kafka Connect and seems to use the standard group.id Kafka Connect setting (group.id=connect-cluster). I have looked into the documentation if there was an option to change this, but I could not find any. Am I missing something or is this currently not configurable?
If this is not configurable, this is a problem in environments in which multiple tenants share the same Kafka cluster. In such an environment, each tenant would need his own Kafka Connect group and topics, such that the tenants do not interfere with each other.
Beat Version: 5.0
Operating System: Linux (SLES12)
Thanks for your reply, Steffen! It's good to know that the Kafka Output is based on Sarama. I was mislead by the "Kafka connect" part of the following error message "Kafka connect fails with: kafka server: The client is not authorized to access this topic."
The client_id setting is not what I am looking for, though.
At the time of that error, the only ACLs I had in place were the following ones, with Kafka configured to allow public access to all topics without ACLs. Without those ACLs in place there is no error when sending data from filebeat.
Do you have an idea why Kafka Output would try to access any of these topics? Unless explicitely configured to send events to one of these topics, of course. But that was not the case.
Current ACLs for resource `Topic:connect-kafka1-status`:
User:CN=kafkaconnectXXX has Allow permission for operations: Read from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: Write from hosts: XXX
Current ACLs for resource `Topic:connect-kafka1-configs`:
User:CN=kafkaconnectXXX has Allow permission for operations: Read from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: Write from hosts: XXX
Current ACLs for resource `Topic:connect-cluster-offsets`:
User:CN=kafkaconnectXXX has Allow permission for operations: Read from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: Write from hosts: XXX
Current ACLs for resource `Group:connect-cluster`:
User:CN=kafkaconnectXXX has Allow permission for operations: All from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: All from hosts: XXX
Current ACLs for resource `Topic:connect-kafka2-status`:
User:CN=kafkaconnectXXX has Allow permission for operations: Read from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: Write from hosts: XXX
Current ACLs for resource `Topic:connect-kafka2-configs`:
User:CN=kafkaconnectXXX has Allow permission for operations: Read from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: Write from hosts: XXX
Current ACLs for resource `Cluster:kafka-cluster`:
User:CN=kafkaXXX has Allow permission for operations: ClusterAction from hosts: XXX
User:CN=kafkaXXX has Allow permission for operations: ClusterAction from hosts: XXX
Current ACLs for resource `Topic:connect-cluster-configs`:
User:CN=kafkaconnectXXX has Allow permission for operations: Read from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: Write from hosts: XXX
Current ACLs for resource `Group:connect-kafka1`:
User:CN=kafkaconnectXXX has Allow permission for operations: All from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: All from hosts: XXX
Current ACLs for resource `Topic:connect-kafka2-offsets`:
User:CN=kafkaconnectXXX has Allow permission for operations: Read from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: Write from hosts: XXX
Current ACLs for resource `Topic:connect-cluster-status`:
User:CN=kafkaconnectXXX has Allow permission for operations: Read from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: Write from hosts: XXX
Current ACLs for resource `Group:connect-kafka2`:
User:CN=kafkaconnectXXX has Allow permission for operations: All from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: All from hosts: XXX
Current ACLs for resource `Topic:connect-kafka1-offsets`:
User:CN=kafkaconnectXXX has Allow permission for operations: Read from hosts: XXX
User:CN=kafkaconnectXXX has Allow permission for operations: Write from hosts: XXX
Are you really sure it's beats accessing these topics? Is kafkaconnect the username you configured in beats (you connect via SASL?)?
I can only imagine (no idea if true or not), kafka brokers accessing some other topics on behalf of clients connecting to a cluster during bootstrapping process. Boostrapping works by clients connecting to one broker and request some 'cluster-state' via GetMetaData API call. The meta-data do contain the list of brokers, topics, partitions, leaders required by clients to finally connect to correct brokers.
Hi, I did some further investigation using a minimal Filebeat 5.0.2 setup and found out that indeed the Kafka output tries to describe all topics present in the cluster when first connecting to the cluster. This leads to the log message "The client is not authorized to access this topic.".
As long as the beat can write to the topic into which it shall forward its events, everything works fine after that, though.
I'll have to check back with the team reporting the problems to see what's different in their setup compared to mine.
Just a minor detail (but to make this more clear), the output itself is not attempting to read these topics. It is kafka itself when trying to serve the GetMetadata API Request. This request is required for correctly bootstrapping the client connecting to the cluster. The request is also executed from time to time in background to find the active leaders, as leader election happens in kafka all the time in background.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.