# Elasticsearch not getting data from logstash

**URL:** <https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166>\
**Category:** Logstash\
**Created:** [June 16, 2018, 7:48am UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166 "2018-06-16T07:48:36Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![rusty\_22](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rusty_22/32/32337_2.png) [@rusty\_22](https://discuss.elastic.co/u/rusty_22)\
**Post date:** [June 16, 2018, 7:48am UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/1 "2018-06-16T07:48:36Z")

</div>

07:37:08.569 [[main]\>worker8] ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error\_message=\>"Elasticsearch Unreachable: [[http://xxxxx:9200/](http://xxxxx:9200/)][Manticore::ClientProtocolException] xxxxx:9200 failed to respond", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will\_retry\_in\_seconds=\>2}  
07:37:10.570 [[main]\>worker8] WARN logstash.outputs.elasticsearch - UNEXPECTED POOL ERROR {:e=\>#\<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections\>}  
07:37:10.570 [[main]\>worker8] ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error\_message=\>"No Available connections", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will\_retry\_in\_seconds=\>4}  
07:37:11.656 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:222] INFO logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://xxxxx:9200/](http://xxxxx:9200/), :path=\>"[http://xxxxxxx:9200](http://xxxxxxx:9200)"}  
07:37:11.738 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.2.6-java/lib/logstash/outputs/elasticsearch/http\_client/pool.rb:222] WARN logstash.outputs.elasticsearch - Restored connection to ES instance {:url=\>#\<URI::HTTP:0x12b1c33 URL:[http://xxxxxx:9200/](http://xxxxxx:9200/)\>}  
08:27:08.568 [[main]\>worker1] WARN logstash.outputs.elasticsearch - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[http://xxxxxx:9200/](http://xxxxxx:9200/)][Manticore::ClientProtocolException] xxxxxx:9200 failed to respond {:url=\>[http://xxxxxx:9200/](http://xxxxxx:9200/), :error\_message=\>"Elasticsearch Unreachable: [[http://xxxxxx:9200/](http://xxxxxx:9200/)][Manticore::ClientProtocolException] xxxxxx:9200 failed to respond", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

getting this exception and it is been occurring since last two days on logstash(connect with elastic cloud) .How can i resolve this

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 16, 2018, 7:50am UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/2 "2018-06-16T07:50:16Z")

</div>

Should you not be using port 9243 with Elastic Cloud instead of 9200? What does your config look like?

---

<div class="post-metadata">

**Author:** ![rusty\_22](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rusty_22/32/32337_2.png) [@rusty\_22](https://discuss.elastic.co/u/rusty_22)\
**Post date:** [June 16, 2018, 7:51am UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/3 "2018-06-16T07:51:15Z")

</div>

Config of which? logstash or elaticsearch

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 16, 2018, 7:52am UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/4 "2018-06-16T07:52:13Z")

</div>

I am looking for the Logstash config, specifically the Elasticsearch output plugin.

---

<div class="post-metadata">

**Author:** ![rusty\_22](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rusty_22/32/32337_2.png) [@rusty\_22](https://discuss.elastic.co/u/rusty_22)\
**Post date:** [June 16, 2018, 7:53am UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/5 "2018-06-16T07:53:47Z")

</div>

if [isMock] == "true" {  
elasticsearch {  
action =\> "index"  
index =\> "mock-%{logType}-%{index\_day}"  
hosts =\> "xxxx:9200"  
user =\> "user"  
password =\> "pass"  
healthcheck\_path =\> "[http://xxxx:9200](http://xxxx:9200)"  
}  
}  
else {  
elasticsearch {  
action =\> "index"  
index =\> "%{logType}-%{index\_day}"  
hosts =\> "xxxx:9200"  
user =\> "user"  
password =\> "pass"  
healthcheck\_path =\> "[http://xxxx:9200](http://xxxx:9200)"  
}  
}

---

<div class="post-metadata">

**Author:** ![rusty\_22](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rusty_22/32/32337_2.png) [@rusty\_22](https://discuss.elastic.co/u/rusty_22)\
**Post date:** [June 16, 2018, 8:00am UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/6 "2018-06-16T08:00:51Z")

</div>

input {  
stdin {  
add\_field =\> {  
"logType" =\> "metric1"  
}  
}  
file {  
path =\> "/data02/logs/samza/samza-container-\*.log"  
add\_field =\> {  
"logType" =\> "samza\_logs"  
}  
}

```
kafka {
   id => "metric2"
   bootstrap_servers => "kafka1:9092,kafka2:9093,kafka3:9094"
   topics => ["metric2"]
   add_field => {
       "logType" => "metric2"
    }
}

kafka {
   id => "metric3"
   bootstrap_servers => "kafka1:9092,kafka2:9093,kafka3:9094"
   topics => ["metric3"]
   add_field => {
       "logType" => "metric3"
    }
}

kafka {
   id => "metric4"
   bootstrap_servers => "kafka1:9092,kafka2:9093,kafka3:9094"
   topics => ["metric4"]
   add_field => {
       "logType" => "metric4"
    }
}

```

}  
filter {

```
if([logType]=="samza_logs") {

    #convert all logstash default field to app field with double underscore from current context

     mutate {
       gsub => [
        "message" , "@timestamp", "timestamp",
        "message" , "_id", "__id",
        "message" , "_index", "__index",
        "message" , "_type", "__type",
        "message" , "_score", "__score"
       ]
    }

    grok { match => { "message" => "%{DATESTAMP:messageTime} \[%{WORD:severity}\] %{DATA:class} \| %{GREEDYDATA:request}"} }   

    if [request] =~ "^\{.*\}[\s\S]*$" { #check whether request is json or not
        json{
            source => "request"
            remove_field => "request"
        } 

        if [mdc] { # check mdc is present in json if then flatting it
          if [mdc][jobName] { mutate { add_field => { "[jobName]" => "%{[mdc][jobName]}" } } }
          if [mdc][jobId] { mutate { add_field => { "[jobId]" => "%{[mdc][jobId]}" } } }
          if [mdc][containerName] { mutate { add_field => { "[containerName]" => "%{[mdc][containerName]}" } } }
          mutate { remove_field => "[mdc]"}
        }
    } 
    if "_jsonparsefailure" in [tags] or "_grokparsefailure" in [tags] { mutate { remove_field => ["request" ,"tags"]} }
    else { mutate { remove_field => ["message"]} }

    if [LogTime] { ruby { code => "event.set('LogTime', DateTime.parse(event.get('LogTime')).strftime('%Y-%m-%dT%H:%M:%S.%L%z'))" } }
    else { ruby { code => "event.set('LogTime', Time.now.strftime('%Y-%m-%dT%H:%M:%S.%L%z'))" }}

    ruby {
        code => "event.set('index_day', Time.now.strftime('%Y-%m-%d'))"
    }
} 

else {
    json {
      source => "message"
      remove_field => "message"
    }

    if [RequestTS] { ruby { code => "event.set('index_day', Date.parse(event.get('RequestTS')).strftime('%Y-%m-%d'))" }}
    else { ruby { code => "event.set('index_day', Time.now.strftime('%Y-%m-%d'))" }}
}

```

}  
output {

```
 if [isMock] == "true" {
    elasticsearch {
    action => "index"
    index => "mock-%{logType}-%{index_day}"
    hosts => "xxxx:9200"
    user => "user"
    password => "pass"
    healthcheck_path => "http://xxxx:9200"
  }
}
else {
  elasticsearch {
    action => "index"
    index => "%{logType}-%{index_day}"
    hosts => "xxxx:9200"
    user => "user"
    password => "pass"
     healthcheck_path => "http://xxxx:9200"
  }
}

```

}

whole configuration

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 16, 2018, 8:55am UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/7 "2018-06-16T08:55:17Z")

</div>

If you are connecting to Elastic Cloud you should use port 9243.

---

<div class="post-metadata">

**Author:** ![rusty\_22](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rusty_22/32/32337_2.png) [@rusty\_22](https://discuss.elastic.co/u/rusty_22)\
**Post date:** [June 16, 2018, 12:18pm UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/8 "2018-06-16T12:18:49Z")

</div>

No it was working previously since last 13 hr it's throwing exception so i think port is not a problem

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 16, 2018, 12:40pm UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/9 "2018-06-16T12:40:25Z")

</div>

Port 9200 is not working on ANY of my Elastic Cloud clusters, so I would recommend changing that to rule it out.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 14, 2018, 12:40pm UTC](https://discuss.elastic.co/t/elasticsearch-not-getting-data-from-logstash/136166/10 "2018-07-14T12:40:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.