The structure I used is filebeat/metricbeat to kafka to logstash to es to kibana.
At the first time logstash actually received data from kafka. However, after I trid to develop the kafka cluster, logstash no longer received anything. I gave up the cluster and returned to previous configure and tried to remove all of the kafka and log (kafka and zookeeper), logstash can not receive anything.
I used kafka-console-consumer.sh check that kafka actually received data from beats, but logstash can not customer the topic.
The version I used is
kafka version I tried include 0.11.0.0 and 0.11.0.3
bootstrap_servers => "myaddress:9092"
topics => ["elk"]
auto_offset_reset => "latest"
consumer_threads => 1
decorate_events => true
follow the office guide https://www.elastic.co/guide/en/beats/metricbeat/current/kafka-output.html
I don't know why It worked at the first time but don't work after I tried to use the cluster.
It is still don't work although I removed kafka and logstash and all of the log and reinstall them.