Fresh install, OutOfMemory error blues


(Dylan Baxter) #1

Fresh install on a VM (CentOS) with 11Gb of RAM, and it doesn't matter what I set Xms/Xmx to I get OutOfMemory. Can someone tell from this log what I am missing?

[2018-05-14T08:15:51,069][INFO ][o.e.n.Node ] [single-test-node] JVM arguments [-Xms8g, -Xmx8g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.Joi5XyYq, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-05-14T08:15:52,134][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [aggs-matrix-stats]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [analysis-common]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [ingest-common]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [lang-expression]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [lang-mustache]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [lang-painless]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [mapper-extras]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [parent-join]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [percolator]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [rank-eval]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [reindex]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [repository-url]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [transport-netty4]
[2018-05-14T08:15:52,135][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [tribe]
[2018-05-14T08:15:52,136][INFO ][o.e.p.PluginsService ] [single-test-node] no plugins loaded
[2018-05-14T08:15:57,056][INFO ][o.e.d.DiscoveryModule ] [single-test-node] using discovery type [zen]
[2018-05-14T08:15:57,977][INFO ][o.e.n.Node ] [single-test-node] initialized
[2018-05-14T08:15:57,979][INFO ][o.e.n.Node ] [single-test-node] starting ...
[2018-05-14T08:15:58,273][INFO ][o.e.t.TransportService ] [single-test-node] publish_address {198.90.22.148:9300}, bound_addresses {198.90.22.148:9300}
[2018-05-14T08:15:58,290][INFO ][o.e.b.BootstrapChecks ] [single-test-node] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-05-14T08:15:58,297][ERROR][o.e.b.Bootstrap ] [single-test-node] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2018-05-14T08:15:58,304][INFO ][o.e.n.Node ] [single-test-node] stopping ...
[2018-05-14T08:15:58,505][INFO ][o.e.n.Node ] [single-test-node] stopped
[2018-05-14T08:15:58,505][INFO ][o.e.n.Node ] [single-test-node] closing ...
[2018-05-14T08:15:58,507][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [single-test-node] fatal error in thread [elasticsearch[single-test-node][generic][T#1]], exiting
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.8.0_171]
at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1025) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]


(Dylan Baxter) #2

I stumbled onto some other people's solution to the system call filters error on CentOS, and disabled the check (though it makes no sense that the last error reported was OutOfMemory). I also set the heap size more appropriately, to 4Gb. The service started! -- for about 20 seconds. Now my log pretty much says the same thing but without the system call filters part:

[2018-05-14T08:30:05,069][INFO ][o.e.n.Node ] [single-test-node] version[6.2.4], pid[9070], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/2.6.32-696.23.1.el6.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b10]
[2018-05-14T08:30:05,069][INFO ][o.e.n.Node ] [single-test-node] JVM arguments [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.ty3qF2PZ, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-05-14T08:30:06,673][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [aggs-matrix-stats]
[2018-05-14T08:30:06,673][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [analysis-common]
[2018-05-14T08:30:06,673][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [ingest-common]
[2018-05-14T08:30:06,673][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [lang-expression]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [lang-mustache]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [lang-painless]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [mapper-extras]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [parent-join]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [percolator]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [rank-eval]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [reindex]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [repository-url]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [transport-netty4]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] loaded module [tribe]
[2018-05-14T08:30:06,674][INFO ][o.e.p.PluginsService ] [single-test-node] no plugins loaded
[2018-05-14T08:30:13,531][INFO ][o.e.d.DiscoveryModule ] [single-test-node] using discovery type [zen]
[2018-05-14T08:30:15,357][INFO ][o.e.n.Node ] [single-test-node] initialized
[2018-05-14T08:30:15,358][INFO ][o.e.n.Node ] [single-test-node] starting ...
[2018-05-14T08:30:15,942][INFO ][o.e.t.TransportService ] [single-test-node] publish_address {198.90.22.148:9300}, bound_addresses {198.90.22.148:9300}
[2018-05-14T08:30:15,954][INFO ][o.e.b.BootstrapChecks ] [single-test-node] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-05-14T08:30:16,094][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [single-test-node] fatal error in thread [elasticsearch[single-test-node][generic][T#2]], exiting
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[?:1.8.0_171]
at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1025) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) ~[?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

Can anyone tell me the true cause?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.