OutOfMemoryError ELK 5.0.2


(Kanagat) #1

Hello guys, today i had problem with ELK.
Next error i had in my log:
2016-12-21T14:21:54,206][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [elasticsearch[node-1][generic][T#8]], exiting
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.8.0_101]
at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]

my server have 32 Gb ram,
and i run there 2 node of ELK
node1 - node.master , Xms1g, Xmx1g
node2 - node.data, Xms15g, Xmx15g.

what kind of recommendation you can give me?
Is it neccesary to low Xms, Xmx for node.data to 13g?


(Mark Walkom) #2

Which node had the error?


(omkar prabhu) #3

Why do you less to node1 ? can you even distrubute around 8GB for each node?

refer this link https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#_give_less_than_half_your_memory_to_lucene


(Kanagat) #4

Data.node:
[2016-12-21T14:11:12,434][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-2] fatal error in thread [elasticsearch[node-2][flush][T#4]], exiting
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.8.0_101]
at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
Master.node:
[2016-12-21T14:21:54,206][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] fatal error in thread [elasticsearch[node-1][generic][T#8]], exiting
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.8.0_101]
at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]


(Kanagat) #5

Both node had the error


(Kanagat) #6

my configuration of jvm data.node:
-Xms13g (before the error there is was 15g)
-Xmx13g (before the error there is was 15g)

GC configuration

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseCompressedOops //this i added earlier

and master node:
-Xms1g
-Xmx1g

GC configuration

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseCompressedOops //this i added earlier

Error occur after my migration from elk 2.34 to 5.0.2.
2 hour earlier i migrate.
at 12:15.


(Kanagat) #7

now i have follow state of nodes :
[event_manager@event-handler-prod1 elasticsearch-5.0.2-node2]$ curl localhost:9212/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
server2 3 61 1 0.00 0.03 0.12 di - node-4
server1 23 61 1 0.09 0.15 0.20 mi - node-1
server1 2 61 1 0.09 0.15 0.20 di - node-2
server2 27 61 1 0.00 0.03 0.12 mi * node-3


(Kanagat) #8

In tomcat i had the same problem.
May be new version of ELK consume more Memory?
21-Dec-2016 14:10:55.703 SEVERE [http-nio-8080-exec-9] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [dispatcher] in context with path [/EventHandlerApp] thre
w exception [Handler processing failed; nested exception is java.lang.OutOfMemoryError: unable to create new native thread] with root cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at org.elasticsearch.threadpool.ThreadPool.(ThreadPool.java:217)
at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:120)
at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:241)
at org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:92)
at org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:81)
at org.elasticsearch.transport.client.PreBuiltTransportClient.(PreBuiltTransportClient.java:71)
at kz.ksp.dsm.spring.elasticsearch.DBConnector.getElasticSearchConn(DBConnector.java:32)


(Kanagat) #9

Warkolm, can you give me some advice?


(Kanagat) #10

it is my app log:
2016-12-21 14:10:25.707 [http-nio-8080-exec-9] DEBUG org.elasticsearch.client.transport.TransportClientNodesService - node_sampler_interval[5s]
2016-12-21 14:10:25.708 [http-nio-8080-exec-9] DEBUG org.elasticsearch.client.transport.TransportClientNodesService - adding address [{#transport#-1}{_gVFiMlPSr6zz2IHygusrQ}{server1}{server1:9300}]
2016-12-21 14:10:25.718 [http-nio-8080-exec-9] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{#transport#-1}{_gVFiMlPSr6zz2IHygusrQ}{server1}{server1:9300}]
2016-12-21 14:10:25.810 [http-nio-8080-exec-9] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{node-1}{oxkDAeF1S1iclLH7sI0PvA}{L4rYBM9IQVeu2hCyOHSzPg}{server1}{server1:9300}]
2016-12-21 14:10:25.810 [http-nio-8080-exec-9] DEBUG org.elasticsearch.client.transport.TransportClientNodesService - adding address [{#transport#-2}{avYl5wWPQia1TmuRKNKLlQ}{server1}{server1:9301}]
2016-12-21 14:10:25.852 [http-nio-8080-exec-9] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{#transport#-2}{avYl5wWPQia1TmuRKNKLlQ}{server1}{server1:9301}]
2016-12-21 14:10:25.883 [http-nio-8080-exec-9] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{node-2}{pyM14CYLS7G1HAViPaHagw}{1XotyG7hTZqa_gV7_RGCzA}{server1}{server1:9301}]
2016-12-21 14:10:25.896 [http-nio-8080-exec-9] DEBUG kz.ksp.dsm.spring.elasticsearch.ConnectionChecker - numberOfDataNodes:1
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - no modules loaded
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty3Plugin]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [force_merge], size [1], queue size [unbounded]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [fetch_shard_started], core [1], max [8], keep alive [5m]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [listener], size [2], queue size [unbounded]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [index], size [4], queue size [200]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [refresh], core [1], max [2], keep alive [5m]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [generic], core [4], max [128], keep alive [30s]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [warmer], core [1], max [2], keep alive [5m]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [search], size [7], queue size [1k]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [flush], core [1], max [2], keep alive [5m]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [fetch_shard_store], core [1], max [8], keep alive [5m]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [management], core [1], max [5], keep alive [5m]
2016-12-21 14:10:55.701 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [get], size [4], queue size [1k]
2016-12-21 14:10:55.702 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [bulk], size [4], queue size [50]
2016-12-21 14:10:55.702 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [snapshot], core [1], max [2], keep alive [5m]
2016-12-21 14:11:13.781 [elasticsearch[client][generic][T#3]] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - disconnecting from [{node-2}{pyM14CYLS7G1HAViPaHagw}{1XotyG7hTZqa_gV7_RG
CzA}{server1}{server1:9301}], channel closed event


(Kanagat) #11

currently i had the same problem only in node2(data-node) and in tomcat:

[2016-12-21T16:45:08,965][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-2] fatal error in thread [elasticsearch[node-2][refresh][T#2]], exiting
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.8.0_101]
at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]

and i noticed that for every query created threadpool. I have checker that every 30 second check my server available and there execute query.

2016-12-21 16:41:41.107 [http-nio-8080-exec-5] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{#transport#-1}{Z95qgsoiQo6uf7Wu3xBmFA}{server1}{server1:9300}]
2016-12-21 16:41:41.196 [http-nio-8080-exec-5] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{node-1}{oxkDAeF1S1iclLH7sI0PvA}{5Yu8N_elR8uv7jmN1bejww}{server1}{1
0.17.9.67:9300}]
2016-12-21 16:41:41.204 [http-nio-8080-exec-5] DEBUG org.elasticsearch.client.transport.TransportClientNodesService - adding address [{#transport#-2}{hoAY9KiiSyWLw_IE_si_AA}{server1}{server1:9301}]
2016-12-21 16:41:41.205 [http-nio-8080-exec-5] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{#transport#-2}{hoAY9KiiSyWLw_IE_si_AA}{server1}{server1:9301}]
2016-12-21 16:41:41.246 [http-nio-8080-exec-5] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{node-2}{pyM14CYLS7G1HAViPaHagw}{QcR0trVVTsS90uZCn0j40Q}{server1}{server1:9301}]
2016-12-21 16:41:41.263 [http-nio-8080-exec-5] DEBUG kz.ksp.dsm.spring.elasticsearch.ConnectionChecker - numberOfDataNodes:1
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - no modules loaded
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty3Plugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [force_merge], size [1], queue size [unbounded]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [fetch_shard_started], core [1], max [8], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [listener], size [2], queue size [unbounded]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [index], size [4], queue size [200]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [refresh], core [1], max [2], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [generic], core [4], max [128], keep alive [30s]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [warmer], core [1], max [2], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [search], size [7], queue size [1k]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [flush], core [1], max [2], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [fetch_shard_store], core [1], max [8], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [management], core [1], max [5], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [get], size [4], queue size [1k]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [bulk], size [4], queue size [50]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [snapshot], core [1], max [2], keep alive [5m]


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.