[Solved] Using the Kafka output the Development Build of Filebeat

Elastic Stack Beats
1

Hello,

I have problems to communicate with my broker as output.
My actually working configuration is :

Filebeat -> Logstash 1 -> RabbitMQ -> Logstash 2 -> Elasticsearch

Logstash1 is only an intermediate between FB and the broker (since Filebeat doesn't support RabbitMQ as output)
Logstash2 is the worker which transform the data

Now , i tried to remove Logstash1 to use only one broker :

Filebeat -> Kafka -> Logstash -> Elasticsearch

But it doesn't seems to work :sweat: , her is my filebeat.yml

filebeat:
  prospectors:
    -
      paths:
        - C://filebeat-1.1.1-windows//logs//*.log*
      input_type: log
output:
    kafka:
    host: "localhost:9092"
    topic: "test"
    use_type: true
    client_id: "beats"
    worker: 1
    max_retries: 0
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 1MB

My message error :

Error Initialising publiser: No outputs are defined. Please define one under the output section.

As i said it works with logtsash or console as output.

My configuration :

Filebeat : filebeat-1.1.1-windows
Kafka : kafka_2.11-0.9.0.1
Java version : 1.7.0_45
Windows : 8.1Entreprise 64bits

Thanks for helping :wink:

2

I see a few issues:

Try this with the development binary:

output:
  kafka:
    hosts: ["localhost:9092"]
    ...
1 Like
3

Thanks a lot, both solutions are working with this new version 1.2.0 :slightly_smiling:

For information my configs files are :

filebeat_kafka.yml

filebeat:
  prospectors:
    -
      paths:
        - C://filebeat//logs//*.log*
      input_type: log
output:
    kafka:
      hosts: ["localhost:9092"]
      topic: "test"
      use_type: true
      client_id: "beats"
      worker: 1
      max_retries: 0
    console:
      pretty: true
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 1MB

filebeat_redis.yml

filebeat:
  prospectors:
    -
      paths:
        - C://filebeat//logs//*.log*
      input_type: log
output:
  redis:
    host: "localhost"
    port: 6379
    save_topology: false
    index: "filebeat"
    db: 99
    db_topology: 1
    timeout: 5
    reconnect_interval: 1
  console:
    pretty: true
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 1MB
4

Hi,
I am using filebeat1.2.3 to push logs to kafka.
From the above kafka configuration due to console: pretty: true, the logs are displying on console but not pushing to kafka topic. If i remove console: pretty: true and run, getting below is message and exited.
filebeat-1.2.3-darwin$ sudo ./filebeat -e -c filebeat-kafka.yml -d "publish"
2016/06/22 10:21:59.311855 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/06/22 10:21:59.312381 publish.go:269: INFO No outputs are defined. Please define one under the output section.
Error Initialising publisher: No outputs are defined. Please define one under the output section.
2016/06/22 10:21:59.312394 beat.go:140: CRIT No outputs are defined. Please define one under the output section.

Filebeat : filebeat-1.1.1-windows
Kafka : kafka_2.11-0.9.0.1
Java version : 1.8.0_71
MAC
Thanks in advance. :slight_smile:

5

kafka output will be shipped with 5.0 release. You can get current 5.0.0-alpha3 from filebeat download page.

6

Yes Steffens, tried with 5.0 release. Its woking. Thanks for the reply.