Filebeat & kafka integration not working

Elastic Stack Beats
1

Hello,

I have probleme to communicate filebeat with Kafka.

My working configuration is :
Filebeat -> Kafka -> Logstash -> Elasticsearch

her is my filebeat.yml

filebeat:
prospectors:
-
paths:
- /appli/okad/logs/*.log
input_type: log
output:
kafka:
host: "localhost:9092"
topic: "test"
use_type: true
client_id: "beats"
worker: 1
max_retries: 0
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 1MB

My configuration :

Filebeat : 1.1.2
Kafka : kafka_2.11-0.8.2.1
Java : 1.8
centos : 7 & 64 bits

I'm testing with a kafka consumer, but the communication does not work, I do not get anything in my Kafka

Thanks for helping

2

Filebeat 1.1.2 does not support Kafka output. This is only available in the nightly builds.

3

Thanks a lot,

how can I install the filebeat that uses Kafka in centos7 ?.

4

You can use a development build:

sudo rpm -ivh https://beats-nightlies.s3.amazonaws.com/filebeat/filebeat-5.0.0-nightlylatest-x86_64.rpm

5

After installing filebeat with the command

I start the process with

filebeat -c /etc/filebeat/filebeat.yml -e -v -d "*"

but I have the following arreur

2016/03/23 15:21:20.783216 kafka.go:183: ERR Invalid kafka configuration: kafka: invalid configuration (Producer.Timeout must be > 0)
2016/03/23 15:21:20.783244 outputs.go:83: ERR failed to initialize kafka plugin as output: kafka: invalid configuration (Producer.Timeout must be > 0)
Start error: fails to load the config: error initializing publisher: kafka: invalid configuration (Producer.Timeout must be > 0)
2016/03/23 15:21:20.783287 beat.go:128: CRIT Start error: fails to load the config: error initializing publisher: kafka: invalid configuration (Producer.Timeout must be > 0)

Thanks for helping

6

Try setting the broker_timeout option. It looks like the default is not being set to 10s as documented.

Related: https://github.com/elastic/beats/issues/1168