Jvm.options with 7.15.0

Elastic Stack Elasticsearch
1

I know the newer version handles ram automatically and that's great.
But I'm seeing the java option sets Xms and Xmx both to 1024m even though I have 96GB of ram on my machine.
Should I just ignore this?

51108 ? SLsl 0:40 /usr/share/Elasticsearch/jdk/bin/ java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -D java .awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -D java .locale.providers=SPI,COMPAT --add-opens= java .base/ java .io=ALL-UNNAMED -XX:+UseG1GC -D java .io.tmpdir=/tmp/Elasticsearch-14653569181007460979 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/Elasticsearch -XX:ErrorFile=/var/log/Elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/Elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Xms1024m -Xmx1024m -XX:MaxDirectMemorySize=536870912 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.path.home=/usr/share/Elasticsearch -Des.path.conf=/etc/Elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -Des.bundled_jdk=true -cp /usr/share/Elasticsearch/lib/* org.Elasticsearch.bootstrap.Elasticsearch -p /var/run/Elasticsearch/Elasticsearch.pid --quiet

51563 pts/1 S+ 0:00 grep --color=auto java

2

Check the heap size (and JVM options) reported in GET _nodes/_all/jvm to see how much heap Elasticsearch thinks it has. If it's wrong, it might be that you're overriding the default behaviour in a config file somewhere, or it might be that Elasticsearch doesn't have access to much of your 96GB of RAM, for instance because it's running in a tiny container.

3

hmmm. It changed to -Xms 31744m now after running for several hours. Would the java option change dynamically?
At least the Xms & Xmx arguments are not in the "input arguments" section.

This is the jvm from one machine.
heap does indeed being set close to 32gb.

      "jvm" : {
        "pid" : 50790,
        "version" : "16.0.2",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "16.0.2+7",
        "vm_vendor" : "Eclipse Foundation",
        "bundled_jdk" : true,
        "using_bundled_jdk" : true,
        "start_time_in_millis" : 1634812026861,
        "mem" : {
          "heap_init_in_bytes" : 33285996544,
          "heap_max_in_bytes" : 33285996544,
          "non_heap_init_in_bytes" : 7667712,
          "non_heap_max_in_bytes" : 0,
          "direct_max_in_bytes" : 0
        },
        "gc_collectors" : [
          "G1 Young Generation",
          "G1 Old Generation"
        ],
        "memory_pools" : [
          "CodeHeap 'non-nmethods'",
          "Metaspace",
          "CodeHeap 'profiled nmethods'",
          "Compressed Class Space",
          "G1 Eden Space",
          "G1 Old Gen",
          "G1 Survivor Space",
          "CodeHeap 'non-profiled nmethods'"
        ],
        "using_compressed_ordinary_object_pointers" : "true",
        "input_arguments" : [
          "-Xshare:auto",
          "-Des.networkaddress.cache.ttl=60",
          "-Des.networkaddress.cache.negative.ttl=10",
          "-XX:+AlwaysPreTouch",
          "-Xss1m",
          "-Djava.awt.headless=true",
          "-Dfile.encoding=UTF-8",
          "-Djna.nosys=true",
          "-XX:-OmitStackTraceInFastThrow",
          "-XX:+ShowCodeDetailsInExceptionMessages",
          "-Dio.netty.noUnsafe=true",
          "-Dio.netty.noKeySetOptimization=true",
          "-Dio.netty.recycler.maxCapacityPerThread=0",
          "-Dio.netty.allocator.numDirectArenas=0",
          "-Dlog4j.shutdownHookEnabled=false",
          "-Dlog4j2.disable.jmx=true",
          "-Djava.locale.providers=SPI,COMPAT",
          "--add-opens=java.base/java.io=ALL-UNNAMED",
          "-XX:+UseG1GC",
          "-Djava.io.tmpdir=/tmp/elasticsearch-2631905464568692967",
          "-XX:+HeapDumpOnOutOfMemoryError",
          "-XX:HeapDumpPath=/var/lib/elasticsearch",
          "-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log",
          "-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m",
          "-Xms31744m",
          "-Xmx31744m",
          "-XX:MaxDirectMemorySize=16642998272",
          "-XX:InitiatingHeapOccupancyPercent=30",
          "-XX:G1ReservePercent=25",
          "-Des.path.home=/usr/share/elasticsearch",
          "-Des.path.conf=/etc/elasticsearch",
          "-Des.distribution.flavor=default",
          "-Des.distribution.type=deb",
          "-Des.bundled_jdk=true"
        ]
      }
4

No, these options are fixed at the time the process starts.

I see them here:

Anyway, this all looks good, so now you can ignore whatever you were seeing before :slight_smile:

5

hmmm. I must be sleeping when posting earlier. :slight_smile:
How does default supposed to work though?
It was not just me seeing -Xms1024m & -Xmx1024m in the java input argument (from ps -ax). There was another engineer debugging with me last night who pointed it out to me. He was about to add jvm.options file in the jvm.options.d directory because of this observation.

Is there a way to know what heap size EX 7.15.0 chose with default when I start the service on the machine itself? Rather than the api call.

I wish to have a way for the script to check locally on the data node when it starts.

Thanks.

By the way, we are not running under a container. It's native app on linux.

6

Another point of confusion, there was a change adding /etc/Elasticsearch/jvm.options.d directory where any .options file can be placed to override the jvm.options file. The doc didn't seem to say it has to have an extension or what it should be, but a file just named "heap" won't work.

7

I think you just create a file jvm.options under the directory and it will be the setting java takes.

8

Just driving by..

There was a time some releases ago when 1gb was the default.. if you have been upgrading without updating the yml(s) for sometimes that could have been a leftover.

9

It's also reported in the startup logs in a couple of places

[2021-10-22T10:33:27,757][INFO ][o.e.n.Node               ] [node-0] JVM arguments [...] -Xms31744m, -Xmx31744m, [...]
                                                                                         ^^^^^^^^^^^^^^^^^^^^^^
...
[2021-10-22T10:33:29,885][INFO ][o.e.e.NodeEnvironment    ] [node-0] heap size [31gb], compressed ordinary object pointers [true]
                                                                                ^^^^

Not really sure why you want to see this without using the API - if a particular heap size is important to you then it's probably best to set it explicitly.

1 Like
10

The reason I want to know it from the node is so the dev_ops could determine if the service has started properly.

We could set it manually just like what we do today with 7.3.1.
But I thought it's recommended to let ES handle it with 7.15.0.
I've read this from several posts regarding breaking changes, and don't configure heap seems to be the overwhelming suggestion.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.