How to connect to Kafka using filebeat with PLAINTEXT SecurityProtocol?

Elastic Stack Beats
1

At present, the security-protocols of kafka mainly include the following PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. But I checked the kafka-output documentation: Configure the Kafka output | Filebeat Reference [8.8] | Elastic, and found that there is no option for PLAINTEXT. How can I use PLAINTEXT to connect to my kafka brokers?

The error I have received so far is: kafka: couldn't fetch broker metadata (check that your client and broker are using the same encryption and authentication settings)

2

What does your filebeat configuration looks like?

PLAINTEXT is the default if you do not configure any security layer on Kafka.

1 Like
3

Thanks, it seems that there are other problems. My configuration is posted below. I can produce and consume messages with this kafka broker address and topic in python.

output.kafka:
  enabled: true
  hosts:
    [
      "10.129.223.255:9094",
      "10.129.191.254:9094",
      "10.129.191.253:9094",
      "10.129.255.251:9094",
    ]
  topic: "test_lwm_log"
  partition.round_robin:
    reachable_only: true    
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000 # 1M
4

I solved it finnaly. We should give a correct version field. In this case, when I use 0.8.2.0, it worked well, while other versions will fail.
In fact, My kafka version is 2.2.1 if I run kafka-broker-api-versions.sh to see the kafka broker version.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.