Filebeat to kafka: Failed to connect to broker

Elastic Stack Beats
1

environment :

  1. kafka 2.11 (installed via ambari)
  2. filebeat 7.4.2 (windows)

according this source since it was installed via ambari i should use port 6667

here is my filebeat.conf

filebeat.inputs:
    - type: log
      paths:
       - C:/Users/A/Desktop/DATA/mailbox3.csv
output.kafka:
  hosts: ["x.x.x.x:6667"]
  topic: "test"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

result

2020-06-10T09:18:01.448+0700    INFO    kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:6667

2020-06-10T09:18:01.450+0700    INFO    kafka/log.go:53 producer/broker/1001 starting up

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [open] on test/0

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 producer/leader/test/0 selected broker 1001

2020-06-10T09:18:01.451+0700    INFO    kafka/log.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host

2020-06-10T09:18:01.452+0700    INFO    kafka/log.go:53 producer/broker/1001 state change to [closing] because dial tcp: lookup x.x.x.x: no such host

2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:264     finished kafka batch
2020-06-10T09:18:01.453+0700    DEBUG   [kafka] kafka/client.go:278     Kafka publish failed with: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.454+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [flushing-3]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [normal]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 state change to [retrying-3]

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/leader/test/0 abandoning broker 1001

2020-06-10T09:18:01.456+0700    INFO    kafka/log.go:53 producer/broker/1001 shut down

this problem similiar to this source but i dont understand how to solve mine.
Thank you

2

Using the Kafka commandline client are you able to connect to x.x.x.x:6667 from the same host where Filebeat is running? Knowing this would help us narrow down the issue.

Shaunak

1 Like
3

Hi, thank you for your respond. Filebeat set on client pc, and kafka is in server pc. I've make sure filebeat able to see kafka pc by pinging client to kafka server

4

It's good that you can ping the kafka server. Next step would be to try and connect to the Kafka broker itself from your client PC, the same one where Filebeat is installed. Kafka provides command-line utilities for this. Can you try that and post the results here, please?

Thanks,

Shaunak

1 Like
5

Dead Shaunak,
thank you so much for your assistance i appreciate it so much, i've solved my problem by doing this:
following this source kafka listeners, since producers came from different environment of kafka server i need to edit server.properties and add advertised.listeners. So i added advertised.listeners the same as listeners which is PLAINTEXT://hostname:6667. It still doesnt work. So i change both of hostname into IP as PLAINTEXT://IP:6667. And it finally works.
Once again thank you so much for your assistance.

Regards,
Charles

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.