Elasticsearch 7.16.2 failed to start when switching from Java 8 to Java 11

Elastic Stack Elasticsearch
1

I'm testing Elasticsearch (ES) single node cluster with Java 8 and Java 11
Java 8 and Java 11 are installed on Linux server RHEL release 7.9 in directories:
/usr/java/jdk1.8.0_281-amd64 includes subdirectories bin, include, jre, legal, lib, man
/usr/java/jdk-11.0.14 includes subdirectories bin, conf, include, jmods, legal, lib, man

Running java -version in cmd returns
java version "1.8.0_281"
Java (TM) SE Runtime Environment (build 1.8.0__281-b09)
Java HotSpot (TM) 64-bit Server VM (build 25.281-b09, mixed mode)

ES single node cluster has been setup and configured with x-pack security from archive package Elasticsearch-7.16.2-linux-x86_64.tar.gz

I'm starting es node with Java 8 (and without using bundled jdk) from launching script as follows

.......................................................  
export PATH=/usr/java/jdk1.8.0_281-amd64/bin
export ES_KEYSTORE_PASSPHRASE_FILE=/path_to_file/es-keystore-sec.txt
cat /path_to_file/es-keystore-sec.txt | ES_JAVA_OPTS=-Des.search.ignore_awareness_attributes=true -Xms4g -Xmx4g bin/elasticsearch -d -p /path_to_pid_file/es-pid > /dev/null 2>&1 &

Node starts gracefully with the appropriate entries in log file when referencing Java 8 in the script

When I switch from Java 8 to Java 11 and changed the 'jdk1.8.0_281-amd64' for 'jdk-11.0.14' in the above script and run it then the node doesn't start and nothing even written in log file

Am I missing something in Java 11 configuration or ES_JAVA_OPTS, or something else need to be changed in the above launching script to conform Java 11?
I noted Java 11 doesn't include JRE. Should I download and install it too?

Any help, suggestion on this issue resolution will be appreciated
Thanks in advance

2

The problem and the solution was in jvm.options default settings for java 9 and higher garbage collection logging

# JDK 9+ GC logging 9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

When I switched the cluster from java 8 to java 11 than the jvm and cluster didn't start with above default setting and the log file even hasn't been created.
The solution was either to specify the full path to gc.log

9-:-Xlog:gc*,gc+age=trace,safepoint:file=/full_path/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

or export environment variable ES_TMPDIR from launching script and specify it in jvm.options

9-:-Xlog:gc*,gc+age=trace,safepoint:file=${ES_TMPDIR}/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

Hope this might help someone with similar problem

1 Like
3

I think it might be reasonable to place a note in elastic docs like "... if you're using java 9+ (or switch your cluster from java 8 to java 9+) and you specify the path to log (not default) in Elasticsearch.yml than you must change in the jvm.options line '9-:-Xlog....' and specify a path to the gc.log with either your full path or with ES_TMPDIR. Othervise after launchig the cluster, the jvm will not start, elastic instance will not start and the log file will not be created at all."
It took me some time to figure this behavior out and might be helpfull for elastic users.

4

Mmmaybe. I mean it's technically a supported configuration, but it's definitely not a recommended one. For instance these docs say:

Modifying advanced settings (i.e. JVM options) is generally not recommended and could negatively impact performance and stability. Using the Elasticsearch-provided defaults is recommended in most circumstances.

and these docs say:

The bundled JVM is the recommended JVM ...

You're doing two not-recommended things by running (a) Java 11 with (b) non-default options. Your problem was really a JDK config issue for which you should consult the docs for your chosen (nonstandard) JDK. Advanced JDK config docs doesn't really seem like something in scope for the Elasticsearch manual.

5

Thanks for the response David.
I got your notes about using bundled java and elastic default settings but still have a few questions about configuration.
I had to use Java 11 to follow up the firm's requirement.

To make the cluster with es-7.16.2 and java 11 functional I had to change the root /config/jvm.options as described above.
I know from docs the changes need to be done in /config/jvm.options.d/jvm.options instead of changing root /config/jvm.options
I tried this use case: left the root /config/jvm.options unchangeable, copied it into /config/jvm.options.d/jvm.options, made the change in it but in this case the node is not starting at all and not writing anything in the log.
Whatever worked for me in the root /config/jvm.options, doesn't work in /config/jvm.options.d/jvm.options

Am I missing something else in configuration ?
Should be file name in directory /config/jvm.options.d/ somehow differ from file name in root directory /config/ ?
Are there any other options how to specify the path to gc.logs instead of changing jvm.options ?

I raised these questions on the forum already at Overriding default jvm options is not working (elastic 7.16.2 and java 11) with no answer yet.
Appreciate your help and attention
Thanks in advance

6

I'd expect problems starting up the JVM to be reported to stderr (perhaps stdout) rather than being written to the log file. We can't write to any log files before starting the JVM.

Elasticsearch should read JVM options from any file whose name ends in .options (i.e. it searches for $ES_CONF/jvm.options.d/*.options). The GC logs are written by the JVM itself so any changes need to be set as a JVM option.

7

Thanks for the response David.
I tested overriding default JVM options on single node cluster and Java 11.
Root jvm options file jvm.options resides in directory /config/
For custom jvm options I tested 2 files - jvm.options and java11-gc-logging.options reside in directory /config/jvm.options.d/

The node started gracefully and the cluster health was green when in root jvm options file /config/jvm.options the default line
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
has been commented out and either in /config/jvm.options.d/jvm.options or in /config/jvm.options.d/java11-gc-logging.options I added the following line
9-:-Xlog:gc*,gc+age=trace,safepoint:file=${ES_TMPDIR}/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
Any others use cases didn't work for me.

Is this normal behavior of overriding default jvm options or I'm still missing something ?

Thanks in advance

8

I would expect the default to work too, but maybe there's something else unusual in your environment that prevents Elasticsearch writing to logs/gc.log? When it didn't work, what did the process report to stderr and stdout?

9

Hi David,

Here is what I did
Didn't change the root jvm options /config/jvm.options and it contains the original line
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m

Created custom jvm options /config/jvm.options.d/gc.options being contained lines

9-:-Xlog:disable
9-:-Xlog:all=warning:stderr:utctime,level,tags
9-:-Xlog:gc*,gc+age=trace,safepoint:file=${ES_TMPDIR}/gc.log:utctime,pid,tags:filecount=32,filesize=64m

The launching script to start the node constructs and exports ${ES_TMPDIR} dynamically pointing to the starting node's logs
The node doesn't start and the log is not even created.

Can you please let me know what I'm doing wrong to being able to write to stdout or/and stderror and which location should I look at ?
Many thanks

10

Sorry, I don't know. I've no idea what could prevent Elasticsearch writing to the console.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.