Hi,
Elastisearch was restarted for some reason and unable to narrow down the reason. Please help.
Logstash
[logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Software caused connection abort: recv failed {:url=>http://127.0.0.1:9200/, :error_message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Software caused connection abort: recv failed", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
ES : GC log
2019-05-31T09:36:44.955+0400: 597837.381: [GC (Allocation Failure) 2019-05-31T09:36:44.955+0400: 597837.381: [ParNew
Desired survivor size 17432576 bytes, new threshold 6 (max 6)
- age 1: 3161688 bytes, 3161688 total
- age 2: 684208 bytes, 3845896 total
- age 3: 929952 bytes, 4775848 total
- age 4: 768664 bytes, 5544512 total
: 286011K->8582K(306688K), 0.0103396 secs] 948944K->671515K(1014528K), 0.0105488 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-05-31T09:36:44.955+0400: 597837.391: Total time for which application threads were stopped: 0.0114828 seconds, Stopping threads took: 0.0000692 seconds
2019-05-31T09:36:45.283+0400: 597837.715: Total time for which application threads were stopped: 0.0007354 seconds, Stopping threads took: 0.0001474 seconds
2019-05-31T09:36:45.283+0400: 597837.716: Total time for which application threads were stopped: 0.0004456 seconds, Stopping threads took: 0.0000981 seconds
2019-05-31T09:36:45.283+0400: 597837.717: Total time for which application threads were stopped: 0.0004534 seconds, Stopping threads took: 0.0001133 seconds
2019-05-31T09:36:45.283+0400: 597837.717: Total time for which application threads were stopped: 0.0004468 seconds, Stopping threads took: 0.0001086 seconds
2019-05-31T09:36:45.283+0400: 597837.718: Total time for which application threads were stopped: 0.0004270 seconds, Stopping threads took: 0.0000884 seconds
2019-05-31T09:36:45.283+0400: 597837.719: Total time for which application threads were stopped: 0.0004307 seconds, Stopping threads took: 0.0000887 seconds
2019-05-31T09:36:45.283+0400: 597837.719: Total time for which application threads were stopped: 0.0004651 seconds, Stopping threads took: 0.0000490 seconds
2019-05-31T09:36:45.283+0400: 597837.720: Total time for which application threads were stopped: 0.0003739 seconds, Stopping threads took: 0.0000348 seconds
2019-05-31T09:36:45.283+0400: 597837.721: Total time for which application threads were stopped: 0.0004918 seconds, Stopping threads took: 0.0001406 seconds
2019-05-31T09:36:45.299+0400: 597837.735: Total time for which application threads were stopped: 0.0010507 seconds, Stopping threads took: 0.0004546 seconds
2019-05-31T09:36:45.299+0400: 597837.736: Total time for which application threads were stopped: 0.0005554 seconds, Stopping threads took: 0.0001502 seconds
2019-05-31T09:36:45.299+0400: 597837.738: Total time for which application threads were stopped: 0.0005942 seconds, Stopping threads took: 0.0002302 seconds
2019-05-31T09:36:45.314+0400: 597837.739: Total time for which application threads were stopped: 0.0006079 seconds, Stopping threads took: 0.0002576 seconds
2019-05-31T09:36:45.314+0400: 597837.741: Total time for which application threads were stopped: 0.0008896 seconds, Stopping threads took: 0.0005027 seconds
2019-05-31T09:36:45.314+0400: 597837.743: Total time for which application threads were stopped: 0.0005201 seconds, Stopping threads took: 0.0001396 seconds
2019-05-31T09:36:45.314+0400: 597837.743: Total time for which application threads were stopped: 0.0005126 seconds, Stopping threads took: 0.0001288 seconds
2019-05-31T09:36:45.314+0400: 597837.744: Total time for which application threads were stopped: 0.0005098 seconds, Stopping threads took: 0.0001331 seconds
2019-05-31T09:36:45.314+0400: 597837.750: Total time for which application threads were stopped: 0.0007317 seconds, Stopping threads took: 0.0003392 seconds
2019-05-31T09:36:45.314+0400: 597837.750: Total time for which application threads were stopped: 0.0004679 seconds, Stopping threads took: 0.0000993 seconds
2019-05-31T09:36:46.267+0400: 597838.696: Total time for which application threads were stopped: 0.0004925 seconds, Stopping threads took: 0.0000984 seconds
2019-05-31T09:36:46.267+0400: 597838.697: Total time for which application threads were stopped: 0.0001924 seconds, Stopping threads took: 0.0000478 seconds
2019-05-31T09:36:46.299+0400: 597838.723: Total time for which application threads were stopped: 0.0003913 seconds, Stopping threads took: 0.0000611 seconds
2019-05-31T09:36:46.439+0400: 597838.865: Total time for which application threads were stopped: 0.0006442 seconds, Stopping threads took: 0.0000779 seconds
2019-05-31T09:36:46.455+0400: 597838.888: Total time for which application threads were stopped: 0.0002771 seconds, Stopping threads took: 0.0001055 seconds
Heap
par new generation total 306688K, used 86572K [0x00000000c0000000, 0x00000000d4cc0000, 0x00000000d4cc0000)
eden space 272640K, 28% used [0x00000000c0000000, 0x00000000c4c29828, 0x00000000d0a40000)
from space 34048K, 25% used [0x00000000d0a40000, 0x00000000d12a1980, 0x00000000d2b80000)
to space 34048K, 0% used [0x00000000d2b80000, 0x00000000d2b80000, 0x00000000d4cc0000)
concurrent mark-sweep generation total 707840K, used 662932K [0x00000000d4cc0000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 66479K, capacity 71915K, committed 72192K, reserved 1112064K
class space used 7918K, capacity 9469K, committed 9508K, reserved 1048576K