Hello!
I have a really strange random behavior since 48 hours, Elasticsearch stopped working.
If I try to restart the service, it is very long to stop it, like it was waiting requests to terminate.
When I finally make it start, API requests don't succeed after a few minutes, and ES become really slow.
I don't have many logs to debug, except some sorts of timeouts:
[2019-01-24T17:42:34,432][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [G8naxcJ] failed to execute on node [G8naxcJ3Tj69X5wRNaaPgA]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [G8naxcJ][127.0.0.1:9300][cluster:monitor/nodes/stats[n]] request_id [141] timed out after [0ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1038) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.4.jar:6.5.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2019-01-24T17:43:02,302][WARN ][o.e.t.TransportService ] [G8naxcJ] Received response for a request that has timed out, sent [54912ms] ago, timed out [54912ms] ago, action [cluster:monitor/nodes/stats[n]], node [{G8naxcJ}{G8naxcJ3Tj69X5wRNaaPgA}{NqbIKoGOSqSleD93u-3sow}{127.0.0.1}{127.0.0.1:9300}{xpack.installed=true}], id [141]
ES is installed on an AWS EC2 instance with 2vCpu and 4Gb of RAM, the heap size is 1Gb, Logstash is installed on the same machine with the same heap size.
There is only one node in the cluster.
I have two indices (filebeat: 2gb/metricbeat: 5gb) of 1 shard each.
Any idea what can cause that timeout? Or where I can find more logs?
Thanks by advance!