Filebeat output connection to Kafka only working on the first execution

[nemoon (Nerea)]

Hello!

I am using Filebeat to stream a log file (PVSS_II.log) to Kafka, but I am having troubles when I execute Filebeat after the first time (it works fine on the first launch).

The first thing that I do is starting my Kafka cluster (1 Zookeeper and 2 brokers), all running in the same Kafka network. Below, I show the ports where each node is listening to: (not showing the complete docker-compose for space issues)

Zookeeper:

networks:
 kafka-network:
  aliases:
  - zookeeper
ports:
- "2181:2181"

Broker1:

networks:
 kafka-network:
  aliases:
  - broker1
ports:
- "9092:9092"	

Broker2:

networks:
 kafka-network:
  aliases:
  - broker2
ports:
- "9093:9092"

The previous Kafka cluster is perfectly running with:

docker-compose -f docker-compose.yml up

Leaving the cluster running on one side, then I used an auxiliar docker container to run Filebeat inside and test if it could send the messages to Kafka, but here it comes the problem.
This is the filebeat.yml that I used, with only Kafka output enabled, so pretty simple configuration:

filebeat.inputs:
- type: log
enabled: true
paths:
- /root/filebeat-7.3.2-linux-x86_64/PVSS_II.log

output.kafka:
enabled: true
hosts: ["broker1:9092"]
topic: "logs_lhc_scada"

Then, I run the command below, which is supposed to send the messages from PVSS_II.log to Kafka as specified in the filebeat.yml config file, and it does, but ONLY for the FIRST TIME I run it:

./filebeat -e -v -c filebeat.yml

Output of the first time I run Filebeat:

(Filebeat finds the file to read and connects to Kafka via already defined brokers)

However, if I stop the execution above ( just Ctrl+C, without exiting the container), and then launching the filebeat command again, I get the following:

Output of the second time I run Filebeat:

As you can see, in the second picture there is no more log message "Connecting to Kafka..." neither "Harvester started for file...", so it is very weird since I didn´t change any configuration in between the 2 executions.

(Say that in the beginning, I was using Kafka 2.3.0 and Filebeat 7.3.2. Then I read Kafka version should be between [0.11 - 2.1.0] for compatibility issues, so I changed to Kafka 2.1.0, but still did not work.)

Anyone could help me or know why this behaviour of Filebeat? I´ve been working on this log file streaming to Kafka with many alternatives for a long time and didn´t find a proper solution yet...

Thanks a lot