Filebeat output connection to Kafka only working on the first execution

Elastic Stack Beats
,
1

Hello!

I am using Filebeat to stream a log file (PVSS_II.log) to Kafka, but I am having troubles when I execute Filebeat after the first time (it works fine on the first launch).

The first thing that I do is starting my Kafka cluster (1 Zookeeper and 2 brokers), all running in the same Kafka network. Below, I show the ports where each node is listening to: (not showing the complete docker-compose for space issues)

Zookeeper:

networks:
 kafka-network:
  aliases:
  - zookeeper
ports:
- "2181:2181"

Broker1:

networks:
 kafka-network:
  aliases:
  - broker1
ports:
- "9092:9092"

Broker2:

networks:
 kafka-network:
  aliases:
  - broker2
ports:
- "9093:9092"

The previous Kafka cluster is perfectly running with:

docker-compose -f docker-compose.yml up

Leaving the cluster running on one side, then I used an auxiliar docker container to run Filebeat inside and test if it could send the messages to Kafka, but here it comes the problem.
This is the filebeat.yml that I used, with only Kafka output enabled, so pretty simple configuration:

filebeat.inputs:
- type: log
enabled: true
paths:
- /root/filebeat-7.3.2-linux-x86_64/PVSS_II.log

output.kafka:
enabled: true
hosts: ["broker1:9092"]
topic: "logs_lhc_scada"

Then, I run the command below, which is supposed to send the messages from PVSS_II.log to Kafka as specified in the filebeat.yml config file, and it does, but ONLY for the FIRST TIME I run it:

./filebeat -e -v -c filebeat.yml

Output of the first time I run Filebeat:

1600×848 409 KB

(Filebeat finds the file to read and connects to Kafka via already defined brokers)

However, if I stop the execution above ( just Ctrl+C, without exiting the container), and then launching the filebeat command again, I get the following:

Output of the second time I run Filebeat:

1600×508 317 KB

As you can see, in the second picture there is no more log message "Connecting to Kafka..." neither "Harvester started for file...", so it is very weird since I didn´t change any configuration in between the 2 executions.

(Say that in the beginning, I was using Kafka 2.3.0 and Filebeat 7.3.2. Then I read Kafka version should be between [0.11 - 2.1.0] for compatibility issues, so I changed to Kafka 2.1.0, but still did not work.)

Anyone could help me or know why this behaviour of Filebeat? I´ve been working on this log file streaming to Kafka with many alternatives for a long time and didn´t find a proper solution yet...

Thanks a lot :slight_smile:

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.