Consuming More RAM Than Configured

[wwalker (Walker)]

Came in this morning to Elasticsearch not running. Checking the event logs, Windows reported Resource-Exhaustion with Elasticsearch consuming 9,965,961,216 bytes (9.2-ish GB) of RAM. JVM Options and the service are both set to use 6GB....so why is ElasticSearch consuming more than it has been given??

[wwalker (Walker)]

Well crap....monitoring tells a different story about usage...any ideas on the discrepancy?

[dadoonet (David Pilato)]

May be your "Windows reported Resource-Exhaustion" reports something else than the HEAP size?

[wwalker (Walker)]

I was under the impression that whatever I set the JVM size to is the extent to which the application consumes. Below is the event that is logged, and was logged many times prior to Elasticsearch going offline. If Elasticsearch consumes more RAM than what you set the JVM size to, how do you accurately size a server? I've got 20GB of RAM with only 12GB allocated to the Elastic Stack (Elasticsearch 6GB, Logstash 5GB, Kibana 1GB). Other than antivirus, nothing else exists on this box, what's consuming the other 7+GB of RAM?

[dadoonet (David Pilato)]

There is something mentioning Virtual Memory. Are you using that? Or did you disable the virtual memory?

Anyway, normally Elasticsearch only consumes what you give to it. But the File system cache can put lot of files in memory as well. May be that's something reported here?

I'm afraid I can't help a lot as I'm not (anymore) a windows user but may be @Martijn_Laarman has an idea?

BTW it's not really recommended to put Logstash and Kibana on the same machine because they will steal some resources from Elasticsearch.

[wwalker (Walker)]

Virtual Memory was turned off when I setup the stack and can confirm that the swap file doesn't exist in the root of C: like it typically does. I forgot that I had MetricBeat collecting stats on the server, here's what was reported just prior to the crash. Any idea what the system.process.memory.size metric is?

[dadoonet (David Pilato)]

More a question to ask on #beats:metricbeat but I guess it's something like the total memory used by all the process? Just a guess so I might be wrong though.

[wwalker (Walker)]

Well crap....opened a thread on Beats and was told it references "The total virtual memory the process has." As shown in an image I previously provided, I have virtual memory disabled (swap file to be technically correct) and I have also configured bootstrap.memory_lock: true which, to my understanding, would prevent Elasticsearch to start if virtual memory was enabled.

[wwalker (Walker)]

This is super frustrating, Elasticsearch won't even stay up for a 24 hour period anymore. Last event I received says that it's consuming over 10GB when it's configured in the service manager for 6GB. These are the settings the service is configured to use, is something wrong here??

-Xms6g
-Xmx6g
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-XX:+HeapDumpOnOutOfMemoryError
-Delasticsearch
-Des.path.home=D:\ElasticStack\Apps\Elasticsearch
-Des.path.conf=D:\ElasticStack\Apps\Elasticsearch\config

Initial memory pool and maximum memory pool are also set to 5120MB. Below is logged seven times around the time that the service stops.

[2018-04-03T19:01:57,003][DEBUG][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [SSSELKLOG.stlouisco.net] caught exception while handling client http traffic, closing connection [id: 0x64e2711d, L:/192.168.1.1:9200 - R:/192.168.1.1:56235]
java.io.IOException: An existing connection was forcibly closed by the remote host
	at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:?]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[?:?]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:?]
	at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) ~[netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

[wwalker (Walker)]

Came in this morning to degraded performance and Kibana reporting over 10k ms response times. Logged into the server and checked RAM usage and below is what I got. Both Elasticsearch and Logstash are configured to use 5GB each, yet the server has almost all 20GB consumed. Private RAM usage is staying within it's bounds like it should, but committed and working are WAY outside of bounds. The server has 20GB of RAM with 11GB set aside for Elastic Stack, why is it consuming the other 9 GB? How do you correctly size a server when this is going on???