Filebeat output connection to Kafka only working on the first execution


I am using Filebeat to stream a log file (PVSS_II.log) to Kafka, but I am having troubles when I execute Filebeat after the first time (it works fine on the first launch).

The first thing that I do is starting my Kafka cluster (1 Zookeeper and 2 brokers), all running in the same Kafka network. Below, I show the ports where each node is listening to: (not showing the complete docker-compose for space issues)


  - zookeeper
- "2181:2181"


  - broker1
- "9092:9092"	


  - broker2
- "9093:9092"

The previous Kafka cluster is perfectly running with:

docker-compose -f docker-compose.yml up

Leaving the cluster running on one side, then I used an auxiliar docker container to run Filebeat inside and test if it could send the messages to Kafka, but here it comes the problem.
This is the filebeat.yml that I used, with only Kafka output enabled, so pretty simple configuration:

- type: log
enabled: true
- /root/filebeat-7.3.2-linux-x86_64/PVSS_II.log

enabled: true
hosts: ["broker1:9092"]
topic: "logs_lhc_scada"

Then, I run the command below, which is supposed to send the messages from PVSS_II.log to Kafka as specified in the filebeat.yml config file, and it does, but ONLY for the FIRST TIME I run it:

./filebeat -e -v -c filebeat.yml

Output of the first time I run Filebeat:

(Filebeat finds the file to read and connects to Kafka via already defined brokers)

However, if I stop the execution above ( just Ctrl+C, without exiting the container), and then launching the filebeat command again, I get the following:

Output of the second time I run Filebeat:

As you can see, in the second picture there is no more log message "Connecting to Kafka..." neither "Harvester started for file...", so it is very weird since I didn´t change any configuration in between the 2 executions.

(Say that in the beginning, I was using Kafka 2.3.0 and Filebeat 7.3.2. Then I read Kafka version should be between [0.11 - 2.1.0] for compatibility issues, so I changed to Kafka 2.1.0, but still did not work.)

Anyone could help me or know why this behaviour of Filebeat? I´ve been working on this log file streaming to Kafka with many alternatives for a long time and didn´t find a proper solution yet...

Thanks a lot :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.