Hi All,
I noticed that one pipeline was falling behind and I checked the logs where I found the following entries, which should mean that there are no available connections or routes to elastic but both instances run in the same server (10.0.0.199).
Any ideas what could cause the lack of connections? Is it the number of the events?
I also see a lot of errors in some JSON messages failing to index due to "JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('"' (code 34)): Expected space separating root-level values"
[2020-02-22T19:50:56,590][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2020-02-22T19:50:56,590][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
[2020-02-22T19:50:57,873][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@10.0.0.199:9200/, :path=>"/"}
[2020-02-22T19:50:57,881][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@10.0.0.199:9200/"}
[2020-02-22T19:53:45,663][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elastic:xxxxxx@10.0.0.199:9200/][Manticore::SocketTimeout] Read timed out {:url=>http://elastic:xxxxxx@10.0.0.199:9200/, :error_message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@10.0.0.199:9200/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2020-02-22T19:53:45,663][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@10.0.0.199:9200/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
Thank you in advance!