currently i had the same problem only in node2(data-node) and in tomcat:
[2016-12-21T16:45:08,965][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-2] fatal error in thread [elasticsearch[node-2][refresh][T#2]], exiting
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.8.0_101]
at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) ~[?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
and i noticed that for every query created threadpool. I have checker that every 30 second check my server available and there execute query.
2016-12-21 16:41:41.107 [http-nio-8080-exec-5] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{#transport#-1}{Z95qgsoiQo6uf7Wu3xBmFA}{server1}{server1:9300}]
2016-12-21 16:41:41.196 [http-nio-8080-exec-5] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{node-1}{oxkDAeF1S1iclLH7sI0PvA}{5Yu8N_elR8uv7jmN1bejww}{server1}{1
0.17.9.67:9300}]
2016-12-21 16:41:41.204 [http-nio-8080-exec-5] DEBUG org.elasticsearch.client.transport.TransportClientNodesService - adding address [{#transport#-2}{hoAY9KiiSyWLw_IE_si_AA}{server1}{server1:9301}]
2016-12-21 16:41:41.205 [http-nio-8080-exec-5] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{#transport#-2}{hoAY9KiiSyWLw_IE_si_AA}{server1}{server1:9301}]
2016-12-21 16:41:41.246 [http-nio-8080-exec-5] DEBUG org.elasticsearch.transport.netty4.Netty4Transport - connected to node [{node-2}{pyM14CYLS7G1HAViPaHagw}{QcR0trVVTsS90uZCn0j40Q}{server1}{server1:9301}]
2016-12-21 16:41:41.263 [http-nio-8080-exec-5] DEBUG kz.ksp.dsm.spring.elasticsearch.ConnectionChecker - numberOfDataNodes:1
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - no modules loaded
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty3Plugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] INFO org.elasticsearch.plugins.PluginsService - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [force_merge], size [1], queue size [unbounded]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [fetch_shard_started], core [1], max [8], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [listener], size [2], queue size [unbounded]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [index], size [4], queue size [200]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [refresh], core [1], max [2], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [generic], core [4], max [128], keep alive [30s]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [warmer], core [1], max [2], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [search], size [7], queue size [1k]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [flush], core [1], max [2], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [fetch_shard_store], core [1], max [8], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [management], core [1], max [5], keep alive [5m]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [get], size [4], queue size [1k]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [bulk], size [4], queue size [50]
2016-12-21 16:42:11.090 [http-nio-8080-exec-9] DEBUG org.elasticsearch.threadpool.ThreadPool - created thread pool: name [snapshot], core [1], max [2], keep alive [5m]