We have Elasticsearch 6.6.2 running on a system with 32 core 128 GB RAM. We have around 400 million data in 10 indices which are running in one node with 5 shards. Also we set 32 GB heap memory for Elasticsearch. Recently we added a new field in index for completion suggester. This additional field results more heap memeory consumption, so we changed the heap space to 64 GB. Even now, the elasticsearch is going down with different exceptions like
org.elasticsearch.transport.RemoteTransportException
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id XXXXX
at org.elasticsearch.search.SearchService.findContext(SearchService.java:598) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$3(SearchService.java:564) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:382) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
.........
.......
.....
org.elasticsearch.transport.RemoteTransportException:
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable on QueueResizingEsThreadPoolExecutor[queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 240.8ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@77990bf4[Running, pool size = 49, active threads = 49, queued tasks = 2028, completed tasks = 52675]]
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-6.6.2.jar:6.6.2]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_161]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:98) ~[elasticsearch-6.6.2.jar:6.6.2]
................
.......
.......
org.elasticsearch.transport.SendRequestTransportException:
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:639) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$1.sendRequest(SecurityServerTransportInterceptor.java:136) ~[?:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:542) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:586) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:577) ~[elasticsearch-6.6.2.jar:6.6.2]
at
...................
...........
....
Caused by: org.elasticsearch.transport.TransportException: TransportService is closed stopped can't send request
Can anybody help to solve this error..