Filebeat send data to kafka

Elastic Stack Beats
1

After installed the filebeat filebeat-5.0.0-alpha1 with filebeat.yml.

filebeat:
  prospectors:
    -
      paths:
        - /var/log/test.log
      input_type: log
output:
  kafka:
    hosts: ["10.120.31.78:9092"]
    topic: "elk"
    use_type: true
    client_id: "beats"
    worker: 1
    max_retries: 0
  console:
    pretty: true
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 1MB

Run the ./filebeat -v -c filebeat.yml successfully.

{
  "@timestamp": "2016-04-07T03:24:55.575Z",
  "beat": {
    "hostname": "peklnelh01",
    "name": "peklnelh01"
  },
  "input_type": "log",
  "message": "9999999",
  "offset": 23,
  "source": "/var/log/test.log",
  "type": "log"
}

But I can not see any data in kafka with ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic elk --from-beginning.
BR

2

What does your filbeat log says?

3

you did set use_type: true, plus you have configued topic. Newer filebeat alpha versions will complain about this being an invalid config, as 2 options to set the kafka topic are configured. Disable use_type option and events should be send to elk topic.