Starting ES-7.8.0 hung up CLI

Elastic Stack Elasticsearch
1

Hello,
I setup ES-7.8.0 cluster on 2 hosts (Linux), 2 master and 2 data nodes on each host.
I'm able to start all 4 nodes, the cluster health=GREEN and I can see all nodes joined the cluster.
Environment: ES-7.8.0 (elasticsearch-7.8.0-linux-x86_64.tar.gz), RHEL 5.11, JDK 1.8.0_65

The problem is when I'm staring a node on CLI then I can see the following two identical lines on the console, where I started ES and these lines hung up CLI

future versions of Elasticsearch will require java 11; your java version from [u/hadoop/java/JDK 1.8.0_65/jre] does not meet this requirement
future versions of Elasticsearch will require java 11; your java version from [u/hadoop/java/JDK 1.8.0_65/jre] does not meet this requirement

To start another node on these console I need to release it manually with Ctl-C, which is not convenient and does Not allow to automate the multi-node startup process on the same console without manual intervention.

I'm starting nodes as follows
ES_JAVA_OPTS="-Dmapper.allow_dots_in_name=true -Xms${ES_HEAP_SIZE} -Xmx${ES_HEAP_SIZE}" bin/elasticsearch -d -p /path_to_pid/es-pid

The ES.log on the second console shows the node is started even the CLI on the first console is not manually released as follows

[WARN][o.e.b.Natives ] node_name unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna/--146.....tmp: /lib64/libc.so.6: version GLIBC_2.7 not found
.................................
... loaded module ...
... loaded module ...
... node_name initialized
... node_name starting ...
..........................
... node_name started

Any help to address this issue will be appreciated
Thanks in advance

2

Can you post more of the log you are seeing please?

3

Thanks for the feedback Mark,

Here are some more information from ES log

[WARN][o.e.b.Natives ] node_name unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/jna/--146.....tmp: /lib64/libc.so.6: version GLIBC_2.7 not found (required by /tmp/jna/--1462778291/jna473135465896030289.tmp)
at java.lang.ClassLoader$NativeLibrary.load(NativeMethod) ~[?1.8.0_65]
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1938) ~[?1.8.0_65]
.................................
at java.lang.Class.forName(Class.java:264) ~[?1.8.0_65]
at org.elasticsearch.bootstrap.Natives.(Natives.java:45) [elasticsearch-7.8.0.jar:7.8.0]
................................
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.8.0.jar:7.8.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.8.0.jar:7.8.0]
[date time goes here][WARN][o.e.b.Natives ] node_name cannot check if running as root because JNA is not available
[date time goes here][WARN][o.e.b.Natives ] node_name cannot register console handler because JNA is not available
[date time goes here][WARN][o.e.b.Natives ] node_name cannot getrlimit RLIMIT_NPROC because JNA is not available
.................................
[date time goes here][WARN][o.e.b.Natives ] node_name version[7.8.0], pid[29283], build[....]
[date time goes here][WARN][o.e.b.Natives ] node_name JVM arguments [...]
[date time goes here][WARN][o.e.b.Natives ] node_name loaded module[aggs-matrix-stats]
.................................
... loaded module ...
... loaded module ...
[date time goes here][WARN][o.e.b.Natives ] node_name initialized
[date time goes here][WARN][o.e.b.Natives ] node_name starting ...
[date time goes here][WARN][o.e.b.Natives ] node_name publish_address ......
..........................
[date time goes here][WARN][o.e.b.Natives ] node_name refreshed keys
[date time goes here][WARN][o.e.b.Natives ] node_name started

The node started and joined the cluster but the starting process capture the CLI console and to release it I need manual intervention with Ctl-C.
I expect the CLI console should be automatically released and available to start another node.

Thanks so much for your help and patience

4

That's pretty standard for Linux based systems. Have a look on the web for running tasks in the background to see what you can do.

Alternatively, you should really use the service files that the DEB/RPM packages provide.

5

Thanks Mark for the feedback.
DEB/RPM packages with service files is not the option for us. We've being running the ES-5.6.3 multi-node cluster in test and prod environments and have the deployment/monitoring process in place already. Now we'd like to migrate the old cluster to new ES-7.8.0.
The starting node command, I provided at the beginning of this thread, worked well with no issues on ES-5.6.3 and that is why I'm wondering why it is Not working on ES-7.8.0.

I'll follow up your advise and search the web.
If you have more specific advise related to ES-7.8.0 starting process and resolving the above mentioned issue it would be appreciated.
Is there any document elsewhere allowing to see/compare/evaluate changes for ES major releases?

6

This is not an Elasticsearch issue. If you run any process in linux in the foreground it'll hold the focus of the shell, then if you use ctrl+c it'll pause the process.

I don't know how you used to start the process, but doing it manually doesn't sound very efficient.

7

Actually I'm running the shell script which at the end produces the following line
ES_JAVA_OPTS="-Dmapper.allow_dots_in_name=true -Xms${ES_HEAP_SIZE} -Xmx${ES_HEAP_SIZE}" bin/elasticsearch -d -p /path_to_pid/es-pid
And this worked for me couple of years with ES-5.6.3. I was able to start first node by running starting script, then the console was released and I was able to start second node from the same terminal and so on.
To start nodes of ES-7.8.0 I'm using the same shell script and the console is not released despite the node started and joined the cluster.

8

Ah ok. Well that's outside the immediate scope of what we can assist with, as it's a custom script.

But if you post the script, we might be able to offer more advice.

9

Thanks Mark for the feedback
The starting process capture the terminal with 2 lines

future versions of Elasticsearch will require java 11; your java version from [u/hadoop/java/JDK 1.8.0_65/jre] does not meet this requirement
future versions of Elasticsearch will require java 11; your java version from [u/hadoop/java/JDK 1.8.0_65/jre] does not meet this requirement

which come up from ES-7.8.0 somehow and that is why I think it is ES-7.8.0 related (?)

10

I think you need to post the entire script you are running.

11

Mark,
[1]I addressed this issue using standard technique and start node as follows
script_to_start_es_node.sh [inp_arg_node_name_to_start] [inp_arg_cluster_name] > /dev/null 2&1 &
and now the node started as expected releasing the cmd and allowing to run the script again to start second node without manual intervention

[2]script_to_start_es_node.sh accepts input arguments, export $JAVA_HOME and at the end produces the following command to start particular node as follows

ES_JAVA_OPTS="-Dmapper.allow_dots_in_name=true -Xms${ES_HEAP_SIZE} -Xmx${ES_HEAP_SIZE}" bin/elasticsearch -d -p /path_to_pid/es-pid > /dev/null 2&1 &

I tried to understand why old version ES-5.6.3 allowed me to start node without the "> /dev/null 2&1 &" and new version ES-7.8.0 starting process captures the cmd terminal and requires Ctl-l to release it.
I think the message on the cmd "future versions of Elasticsearch will require java 11; your java version from [u/hadoop/java/JDK 1.8.0_65/jre] does not meet this requirement" has been produced by the ES-7.8.0 code. It seems the ES-7.8.0 checks up the java version somehow (if bundled jdk is not in use) and produces the alert message about the "...java 11..." and captures the cmd terminal, which, I think, shouldn't happen (?)
Thanks

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.