Filebeat unable to output logs to kafka

Elastic Stack Beats
1

I had setup a 3 node clusters and my kafka / zookeeper are running
i can create a topic and insert msg to the topic and read its output

  • filebeat version 7.12.0 (amd64), libbeat 7.12.0
  • kafka 2.13-2.7.0
  • zookeeper 3.5.8

However, filebeat is not able to output to kafka.

Here's my filebeat config
filebeat.inputs:

  • type: log
    enabled: true

  • /xxxx/xxxxx/xxxxxx/xxx/xxxx.txt

filebeat.config.modules:

Glob pattern for configuration loading

path: ${path.config}/modules.d/*.yml

Set to true to enable config reloading

reload.enabled: true

Period on which files under path should be checked for changes

reload.period: 10s

output.kafka:

enabled: true

hosts: ["192.100.100.120:9092", "192.100.100.122:9092", "192.100.100.123:9092"]

topic: TutorialTopic
#topic: %{[TutorialTopic]}

Authentication details. Password is required if username is set.

username: 'xxxxxx'
password: 'xxxxxx'

i can start filebeat and from the logs

"mac": [
"00:0c:29:02:xx:xx",
"00:0c:29:02:xx:xx"
],
"hostname": "xxxxxxx"
}
}
2021-07-11T21:51:04.644+0800 DEBUG [harvester] log/log.go:107 End of file reached: /xxxxxx.txt; Backoff now.

i do have 1 error
ERROR [kafka] kafka/client.go:317 Kafka (topic=TutorialTopic): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Any advises are deeply appreciated.

2

This means that filebeat cannot communicate with your Kafka Brokers.

Maybe the machine running filebeat cannot talk with the broker nodes or your Kafka configuration for listeners and advertised listeners is wrong.

Can you try to create a topic from the same machine that is running filebeat using the Kafka console scripts and using the same endpoints?

3

Hi Leandro,
the server running filebeat are running low on resources. Is there any way to add messages to the topic without installing kafka?

4

i test the port via nc and its good
[root@:XXX102 filebeat]# nc -zv 192.100.100.120 9092
Ncat: Version 7.50 ( Ncat - Netcat for the 21st Century )
Ncat: Connected to 192.100.100.120:9092.
Ncat: 0 bytes sent, 0 bytes received in 0.02 seconds.

i can list its brokers and topic
[root@XKAF101 bin]# ~/kafka/bin/zookeeper-shell.sh XKAF101:2181 ls /brokers/ids

Connecting to XKAF101:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[1, 2, 3]

[root@XKAF101 bin]# ~/kafka/bin/zookeeper-shell.sh XKAF101:2181 ls /brokers/topics

Connecting to XKAF101:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[TutorialTopic, __consumer_offsets]

i wonder what's wrong and i did a tcp dump and the packets hit the kafka server #1

22:46:09.151310 00:0c:29:f2:82:30 (oui Unknown) > 00:0c:29:2e:30:b2 (oui Unknown), IPv4, length 128: XKAF102.42938 > XKAF101.XmlIpcRegSvc: tcp 62
0x0000: 4500 0072 ab22 4000 4006 45a8 c064 647a E..r."@.@.E..ddz
0x0010: c064 6478 a7ba 2384 ee8e 706e 9407 e5d8 .ddx..#...pn....
0x0020: 8018 3908 9c69 0000 0101 080a 0012 14aa ..9..i..........
0x0030: 0014 1dc9 0000 003a 0001 000c 0000 08c3 .......:........
0x0040: 0012 6272 6f6b 6572 2d32 2d66 6574 6368 ..broker-2-fetch
0x0050: 6572 2d30 0000 0000 0200 0001 f400 0000 er-0............
0x0060: 0100 a000 0000 442e a41c 0000 08c2 0101 ......D.........

5

i tested 3 hosts via another server and the brokers are up
i believed it is due to some configurations which either on the kafka or filebeat which i'm not able to figure it up.
[root@kafdrop /]kafkacat -Lb XKAF101:9092
Metadata for all topics (from broker 1: XKAF101:9092/1):
3 brokers:
broker 2 at XKAF102:9092
broker 3 at XKAF103:9092
broker 1 at XKAF101:9092 (controller)
4 topics:
topic "TutorialTopic" with 1 partitions:
partition 0, leader 2, replicas: 2,1, isrs: 2,1
topic "testtest" with 3 partitions:
partition 0, leader 1, replicas: 1,3, isrs: 3,1
partition 1, leader 2, replicas: 2,1, isrs: 2,1
partition 2, leader 3, replicas: 3,2, isrs: 2,3
topic "test" with 3 partitions:
partition 0, leader 2, replicas: 2,3, isrs: 3,2
partition 1, leader 3, replicas: 3,1, isrs: 3,1
partition 2, leader 1, replicas: 1,2, isrs: 2,1
topic "__consumer_offsets" with 3 partitions:
partition 0, leader 1, replicas: 1,2, isrs: 2,1
partition 1, leader 2, replicas: 2,3, isrs: 3,2
partition 2, leader 3, replicas: 3,1, isrs: 3,1

6

From the filebeat server test the connection to all of your brokers, use the kafka console scripts, you don't need to run kafka on this server, just use the scripts to publish and consume from your brokers.

If this works without any problem, change the filebeat log level to debug and see if gives you more information.

Also enable the debug logs in your kafka cluster and see if this give you more information.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.