Can't get event when using Kafka with logstash

Elastic Stack Logstash
1

Hi,
i'm using ES, LS, Kibana 5.3, beat 5.4 and Kafka 0.10.2
I used winlogbeat to send event log to ES

Beat -> Logstash collector -> Kafka -> Logstash Processor -> ElasticSearch

This is my config in logstash collector

input {
  beats {
    port => 5044
  }
}
filter {
  if [type] == "wineventlog" and [event_id] == 5156 {
    drop { }
  }
  mutate {
   add_field => {"beatname" => "%{[@metadata][beat]}"}
   add_field => {"beattype" => "%{[@metadata][type]}"}
  }
}
output {
        kafka {
                bootstrap_servers => "kafka-01:9092,kafka-02:9093,kafka-03:9094"
                codec => plain {
                        format => "%{message}"
                }
                topic_id => "logstash-log"
        }
}

This is my config in logstash processor

input {
        kafka {
                bootstrap_servers => "kafka-01:9092,kafka-02:9093,kafka-03:9094"
                consumer_threads => 8
                group_id => "logstash"
                topics => "logstash-log"
        }
}
output {
        elasticsearch {
                hosts => ["https://ES-02:9200", "https://ES-03:9200", "https://ES-04:9200"]
                user => "elastic"
                cacert => "/etc/logstash/ca.pem"
                ssl => true
                password => "mypassword"
                manage_template => false
                flush_size => 5000
                index => "%{beatname}-%{+xxxx.ww}"
                document_type => "%{beattype}"
        }
}

im using Kafka Ui tool for monitor multi-broker cluster

I checked in tool, no new message in Kafka, no consumer .... After that i checkd in /var/log/logstash/logstash-plain.log, .. no error.
I try to set output is file, got many event log ..

So what happen? Something wrong with my logstash config.

Thansk

2

Hey, I've been having the same issue. I was told that 0.10.1.x is not compatible with LS 5.x . So I installed 6.2.7 version of logstash Plugin for both Input and Output and still it doesn't work for me. I've tried everything.

Logstash -> Kafka -> ELK

Follow my thread, may be it will be useful to troubleshoot your issue.

3

were you able to get this working?. I'm curious as it can help me solving my issue.

5

I read some thread in this forum, they said some version of kafka plugin not support 0.10.1 or 0.10.2 ... wait developer fix it ...
Now i used rabbitmq . It's simple and very easy config. If u need help with rabbitmq, i can help u :slight_smile:

6

Thank you. I will look into rabbitmq as well. Appreciate your response.

7

Hey , I got mine working. see below my application stack. Also, make sure that port 9092 Ingress is open on your Kafka Cluster from ELK servers. I switched my logging level to 'debug' which helped figure out the issue.

Kafka - 2.11_0.10.1
Zookeeper - 3.4.9-1
ES - 5.0.1
Kibana - 5.0.1
LS - 5.1.1
logstash-input-kafka - 6.2.7
​​​​​​​logstash-output-kafka - 5.1.1