Logstash Error- Failing to send monitoring events

Dear All,

Trying to trouble shoot the logstash issues that I'm facing, and any hints would be helpful

Background
Note the issue is in both prod and dev, and the numbers below are from dev
Log ES Version : 6.2.1
Monitoring ES Version : 6.3.0
Logstash version: docker.elastic.co:443/logstash/logstash 6.2.4

Both logging elastic, and logstash send monitoring data to Monitoring ES.

Getting the below error after sometime (both in initial case and also after restart). But monitoring cluster is working and good. Restarting the logstash send events for a while, but comes back with same state after sometime. Observing the log, I could find host_not_found for monitoring cluster. But monitoring keeps emitting the log, and not sending logstash monitoring events.

[WARN ] 2018-12-10 20:02:49.660 [Ruby-0-Thread-9@[.monitoring-logstash]>worker0: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385] elasticsearch - Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://logstash_user2:xxxxxx@esmonitoring.........:9200/][Manticore::SocketTimeout] Read timed out {:url=>https://logstash_user2:xxxxxx@esmonitoring.........:9200/, :error_message=>"Elasticsearch Unreachable: [https://logstash_user2:xxxxxx@esmonitoring..........:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[ERROR] 2018-12-10 20:02:49.666 [Ruby-0-Thread-9@[.monitoring-logstash]>worker0: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385] elasticsearch - Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://logstash_user2:xxxxxx@esmonitoring..........:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[WARN ] 2018-12-10 20:02:51.684 [Ruby-0-Thread-9@[.monitoring-logstash]>worker0: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385] elasticsearch - UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[ERROR] 2018-12-10 20:02:51.688 [Ruby-0-Thread-9@[.monitoring-logstash]>worker0: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385] elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
[ERROR] 2018-12-10 20:03:56.089 [pool-2-thread-1] metrics - Failed to create monitoring event {:message=>"undefined method `system?' for nil:NilClass", :error=>"NoMethodError"} (Repeats)

NOTE: Monitoring cluster is health, and hardly gets around 30 events/sec..It is a 3 Node cluster just capturing only monitoring data.

Any help/guidance would be helpful.

Rechecking as I'm still facing the issue :-(. Any guidance would be helpful

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.