# Filebeat output connection to Kafka only working on the first execution

**URL:** <https://discuss.elastic.co/t/filebeat-output-connection-to-kafka-only-working-on-the-first-execution/200523>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [September 21, 2019, 11:19am UTC](https://discuss.elastic.co/t/filebeat-output-connection-to-kafka-only-working-on-the-first-execution/200523 "2019-09-21T11:19:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nemoon](https://avatars.discourse-cdn.com/v4/letter/n/4bbf92/32.png) [@nemoon](https://discuss.elastic.co/u/nemoon)\
**Post date:** [September 21, 2019, 11:19am UTC](https://discuss.elastic.co/t/filebeat-output-connection-to-kafka-only-working-on-the-first-execution/200523/1 "2019-09-21T11:19:57Z")

</div>

Hello!

I am using Filebeat to stream a log file (`PVSS_II.log`) to Kafka, but I am having troubles when I execute Filebeat after the first time (it works fine on the first launch).

The first thing that I do is starting my Kafka cluster (1 Zookeeper and 2 brokers), all running in the same Kafka network. Below, I show the ports where each node is listening to: (not showing the complete docker-compose for space issues)

Zookeeper:

```
networks:
 kafka-network:
  aliases:
  - zookeeper
ports:
- "2181:2181"

```

Broker1:

```
networks:
 kafka-network:
  aliases:
  - broker1
ports:
- "9092:9092"	

```

Broker2:

```
networks:
 kafka-network:
  aliases:
  - broker2
ports:
- "9093:9092"

```

The previous Kafka cluster is perfectly running with:

> docker-compose -f docker-compose.yml up

Leaving the cluster running on one side, then I used an auxiliar docker container to run Filebeat inside and test if it could send the messages to Kafka, but here it comes the problem.  
This is the `filebeat.yml` that I used, with only Kafka output enabled, so pretty simple configuration:

> filebeat.inputs:  
> - type: log  
> enabled: true  
> paths:  
> - /root/filebeat-7.3.2-linux-x86\_64/PVSS\_II.log

> output.kafka:  
> enabled: true  
> hosts: ["broker1:9092"]  
> topic: "logs\_lhc\_scada"

Then, I run the command below, which is supposed to send the messages from `PVSS_II.log` to Kafka as specified in the `filebeat.yml` config file, and it does, but ONLY for the FIRST TIME I run it:

> ./filebeat -e -v -c filebeat.yml

Output of the first time I run Filebeat:

 ![](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5fcc4a1024a9b659fbba3acb77974d5b1d506246.jpeg)  
(Filebeat finds the file to read and connects to Kafka via already defined brokers)

However, if I stop the execution above ( just Ctrl+C, without exiting the container), and then launching the filebeat command again, I get the following:

Output of the second time I run Filebeat:

 ![](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af39fb72e9285007c2232170b3d8c8cdaa54b46e.jpeg)  
As you can see, in the second picture there is no more log message "Connecting to Kafka..." neither "Harvester started for file...", so it is very weird since I didn´t change any configuration in between the 2 executions.

(Say that in the beginning, I was using Kafka 2.3.0 and Filebeat 7.3.2. Then I read Kafka version should be between [0.11 - 2.1.0] for compatibility issues, so I changed to Kafka 2.1.0, but still did not work.)

Anyone could help me or know why this behaviour of Filebeat? I´ve been working on this log file streaming to Kafka with many alternatives for a long time and didn´t find a proper solution yet...

Thanks a lot 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2019, 11:19am UTC](https://discuss.elastic.co/t/filebeat-output-connection-to-kafka-only-working-on-the-first-execution/200523/2 "2019-10-19T11:19:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.