Hello!
Issue:
Usually everything works correctly. Logstash succsessfuly writes bulk requests with events to Elasticsearch. But sometimes we have issue with connectivity between Logstash && Elasticsearch. Issue relates to bulk write errors. There are few error messages in Logstash logs, but log of Elasticsearch is quiet.
But in logs of Kibana - we found some logs about bulk errors too.
We're testing connectivity between Logstash and Elastic - there are no problems there. No packet drops or something else.
Question:
What is this problem about? And how we can debug it more?
Error messages from Logstash:
logstash | 2022-11-21T09:40:29.048459401Z [2022-11-21T12:40:29,048][WARN ][logstash.outputs.elasticsearch][PIPELINE][dd90558acd8d02c476931717c3913a4a5087b2c57ebb69c24c99de2ff16f4e4c] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://<ELK NODE>:9200/_bulk][Manticore::SocketTimeout] Read timed out {:url=>https://elastic:xxxxxx@<ELK NODE>:9200/, :error_message=>"Elasticsearch Unreachable: [https://<ELK NODE>:9200/_bulk][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash | 2022-11-21T09:40:59.204142343Z [2022-11-21T12:40:59,203][ERROR][logstash.outputs.elasticsearch][PIPELINE][cd63ee8d7d6a1be90ac459cae26b964a203ec293b880fe679b574af148a8e486] Attempted to send a bulk request but Elasticsearch appears to be unreachable or down {:message=>"Elasticsearch Unreachable: [https://<ELK NODE>:9200/_bulk][Manticore::SocketTimeout] Read timed out", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :will_retry_in_seconds=>64}
Errors message from Kibana:
kibana_1 | {"type":"log","@timestamp":"2022-11-21T12:31:42+03:00","tags":["error","plugins","eventLog"],"pid":7,"message":"error writing bulk events: \"Request timed out\"; docs: [{\"create\":{\"_index\":\".kibana-event-log-7.16.3\"}},{\"@timestamp\":\"2022-11-21T09:29:41.429Z\",\"event\":{\"provider\":\"actions\",\"action\":\"execute\",\"start\":\"2022-11-21T09:27:41.416Z\",\"end\":\"2022-11-21T09:29:41.429Z\",\"duration\":120013000000,\"outcome\":\"failure\"},\"kibana\":{\"task\":{\"scheduled\":\"2022-11-21T09:26:45.070Z\",\"schedule_delay\":56346000000},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"action\",\"id\":\"b6bd78f0-8347-11ec-9bd8-49a6783d58be\",\"type_id\":\".index\",\"namespace\":\"space1\"},{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"0ec93b10-83de-11ec-9bd8-49a6783d58be\",\"type_id\":\".index-threshold\",\"namespace\":\"space1\"}],\"server_uuid\":\"8373c110-b823-4cc2-8f68-d57827dbb0d5\",\"version\":\"7.16.3\"},\"message\":\"action execution failure: .index:b6bd78f0-8347-11ec-9bd8-49a6783d58be: Detect Loss\",\"error\":{\"message\":\"error indexing documents: Request timed out\"},\"ecs\":{\"version\":\"1.8.0\"}},{\"create\":{\"_index\":\".kibana-event-log-7.16.3\"}},{\"@timestamp\":\"2022-11-21T09:29:41.436Z\",\"event\":{\"provider\":\"actions\",\"action\":\"execute\",\"start\":\"2022-11-21T09:27:41.425Z\",\"end\":\"2022-11-21T09:29:41.436Z\",\"duration\":120011000000,\"outcome\":\"failure\"},\"kibana\":{\"task\":{\"scheduled\":\"2022-11-21T09:26:50.819Z\",\"schedule_delay\":50606000000},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"action\",\"id\":\"b6bd78f0-8347-11ec-9bd8-49a6783d58be\",\"type_id\":\".index\",\"namespace\":\"space1\"},{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"30e8b190-83dd-11ec-9bd8-49a6783d58be\",\"type_id\":\".index-threshold\",\"namespace\":\"space1\"}],\"server_uuid\":\"8373c110-b823-4cc2-8f68-d57827dbb0d5\",\"version\":\"7.16.3\"},\"message\":\"action execution failure: .index:b6bd78f0-8347-11ec-9bd8-49a6783d58be: Detect Loss\",\"error\":{\"message\":\"error indexing documents: Request timed out\"},\"ecs\":{\"version\":\"1.8.0\"}},{\"create\":{\"_index\":\".kibana-event-log-7.16.3\"}},{\"@timestamp\":\"2022-11-21T09:29:41.469Z\",\"event\":{\"provider\":\"actions\",\"action\":\"execute\",\"start\":\"2022-11-21T09:27:41.458Z\",\"end\":\"2022-11-21T09:29:41.469Z\",\"duration\":120011000000,\"outcome\":\"failure\"},\"kibana\":{\"task\":{\"scheduled\":\"2022-11-21T09:26:48.120Z\",\"schedule_delay\":53338000000},\"saved_objects\":[{\"rel\":\"primary\",\"type\":\"action\",\"id\":\"b6bd78f0-8347-11ec-9bd8-49a6783d58be\",\"type_id\":\".index\",\"namespace\":\"space1\"},{\"rel\":\"primary\",\"type\":\"alert\",\"id\":\"279f7f80-83e0-11ec-9bd8-49a6783d58be\",\"type_id\":\".index-threshold\",\"namespace\":\"space1\"}],\"server_uuid\":\"8373c110-b823-4cc2-8f68-d57827dbb0d5\",\"version\":\"7.16.3\"},\"message\":\"action execution failure: .index:b6bd78f0-8347-11ec-9bd8-49a6783d58be: Detect Loss\",\"error\":{\"message\":\"error indexing documents: Request timed out\"},\"ecs\":{\"version\":\"1.8.0\"}}]"}
More:
Docker Engine ver. - 20.10.21
Elastic ver. - 7.16.3
Logstash ver. - 7.17.2