shjdwxy
April 21, 2019, 9:31am
1
hi,
ES was running normally and GC is normal too. But one node suddenly OOM with "fatal error on the network layer"
logs link:
gistfile1.txt
[2019-04-21T12:55:59,254][WARN ][i.n.c.AbstractChannelHandlerContext] An exception 'java.lang.OutOfMemoryError: unable to create new native thread' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.8.0_162]
at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) ~[?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) ~[?:1.8.0_162]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-5.4.3.jar:5.4.3]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-5.4.3.jar:5.4.3]
at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1512) ~[elasticsearch-5.4.3.jar:5.4.3]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1396) ~[elasticsearch-5.4.3.jar:5.4.3]show original
Does anyone meet this before? How to debug this problem ?
Thank you in advance!
By default Elasticsearch will write a heap dump when it encounters an OutOfMemoryError. The best thing to do is to open this heap dump (e.g. in MAT ) and investigate what was using all the heap.
system
May 19, 2019, 12:04pm
4
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.