# Filebeat & kafka integration not working

**URL:** <https://discuss.elastic.co/t/filebeat-kafka-integration-not-working/45207>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 23, 2016, 10:40am UTC](https://discuss.elastic.co/t/filebeat-kafka-integration-not-working/45207 "2016-03-23T10:40:42Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![rkarra](https://avatars.discourse-cdn.com/v4/letter/r/ebca7d/32.png) [@rkarra](https://discuss.elastic.co/u/rkarra)\
**Post date:** [March 23, 2016, 10:40am UTC](https://discuss.elastic.co/t/filebeat-kafka-integration-not-working/45207/1 "2016-03-23T10:40:42Z")

</div>

Hello,

I have probleme to communicate filebeat with Kafka.

My working configuration is :  
Filebeat -\> Kafka -\> Logstash -\> Elasticsearch

her is my filebeat.yml

> filebeat:  
> prospectors:  
> -  
> paths:  
> - /appli/okad/logs/\*.log  
> input\_type: log  
> output:  
> kafka:  
> host: "localhost:9092"  
> topic: "test"  
> use\_type: true  
> client\_id: "beats"  
> worker: 1  
> max\_retries: 0  
> shipper:  
> logging:  
> files:  
> rotateeverybytes: 10485760 # = 1MB

My configuration :

> Filebeat : 1.1.2  
> Kafka : kafka\_2.11-0.8.2.1  
> Java : 1.8  
> centos : 7 & 64 bits

I'm testing with a kafka consumer, but the communication does not work, I do not get anything in my Kafka

Thanks for helping

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 23, 2016, 12:32pm UTC](https://discuss.elastic.co/t/filebeat-kafka-integration-not-working/45207/2 "2016-03-23T12:32:32Z")

</div>

Filebeat 1.1.2 does not support Kafka output. This is only available in the nightly builds.

---

<div class="post-metadata">

**Author:** ![rkarra](https://avatars.discourse-cdn.com/v4/letter/r/ebca7d/32.png) [@rkarra](https://discuss.elastic.co/u/rkarra)\
**Post date:** [March 23, 2016, 1:04pm UTC](https://discuss.elastic.co/t/filebeat-kafka-integration-not-working/45207/3 "2016-03-23T13:04:26Z")

</div>

Thanks a lot,

how can I install the filebeat that uses Kafka in centos7 ?.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 23, 2016, 1:21pm UTC](https://discuss.elastic.co/t/filebeat-kafka-integration-not-working/45207/4 "2016-03-23T13:21:18Z")

</div>

You can use a development build:

`sudo rpm -ivh https://beats-nightlies.s3.amazonaws.com/filebeat/filebeat-5.0.0-nightlylatest-x86_64.rpm`

---

<div class="post-metadata">

**Author:** ![rkarra](https://avatars.discourse-cdn.com/v4/letter/r/ebca7d/32.png) [@rkarra](https://discuss.elastic.co/u/rkarra)\
**Post date:** [March 23, 2016, 3:31pm UTC](https://discuss.elastic.co/t/filebeat-kafka-integration-not-working/45207/5 "2016-03-23T15:31:45Z")

</div>

After installing filebeat with the command

I start the process with

> filebeat -c /etc/filebeat/filebeat.yml -e -v -d "\*"

but I have the following arreur

> 2016/03/23 15:21:20.783216 kafka.go:183: ERR Invalid kafka configuration: kafka: invalid configuration (Producer.Timeout must be \> 0)  
> 2016/03/23 15:21:20.783244 outputs.go:83: ERR failed to initialize kafka plugin as output: kafka: invalid configuration (Producer.Timeout must be \> 0)  
> Start error: fails to load the config: error initializing publisher: kafka: invalid configuration (Producer.Timeout must be \> 0)  
> 2016/03/23 15:21:20.783287 beat.go:128: CRIT Start error: fails to load the config: error initializing publisher: kafka: invalid configuration (Producer.Timeout must be \> 0)

Thanks for helping

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 23, 2016, 5:36pm UTC](https://discuss.elastic.co/t/filebeat-kafka-integration-not-working/45207/6 "2016-03-23T17:36:49Z")

</div>

Try setting the [`broker_timeout`](https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html#_broker_timeout) option. It looks like the default is not being set to 10s as documented.

Related: [https://github.com/elastic/beats/issues/1168](https://github.com/elastic/beats/issues/1168)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:54pm UTC](https://discuss.elastic.co/t/filebeat-kafka-integration-not-working/45207/7 "2017-07-05T21:54:14Z")

</div>


