Logstash stops reading from Kafka

Logstash: 6.3.2-1

       bootstrap_servers => 'kafka-end-point:9092'
       topics => ["topic1", "topic2", "topic3", "topic4","topic5","topic6","topic7","topic8"]
       codec => "json"
       group_id => "logstash-ec2-2"
       consumer_threads => "60"
       decorate_events => true

    filter {
            json {
             source => "message"
             #target => "appmessage"
            json {
             source => "msg"
             #target => "[message][atts]"

                    elasticsearch {
                           hosts => ["http://es-node:9200"]
                           index => "%{[@metadata][kafka][topic]}-%{+YYYY.MM.dd}"
                           codec => "json"
                           sniffing => false

It runs for some time without issues and then stops reading logs from Kafka

We get this error

    [2018-10-25T12:17:08,405][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://es-node:9200/][Manticore::ClientProtocolException] es-nodes.production.pm.com:9200 failed to respond", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2018-10-25T12:17:08,428][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}

I have verified that elasticsearch is accessible from logstash machines

We have 13 nodes of ES fronted by an ELB(AWS). All of them are healthy

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.