Elasticsearch v7.6.2 Failed to Start, Killed by SIGABRT on RHEL 7.7 - [URGENT]

Black-Hat · May 4, 2020, 9:04pm

Hi All,

I'm having issue to start elastic services with error of:

    Job for elasticsearch.service failed because a fatal signal was delivered to the control process.

This is a completely new setup and I tried to configure elasticsearch.yml accordingly but failed to start the services. I even tried to start the services without editing any configuration, but still no luck to bring up the services.

Journalctl -xe provide info below:

    Process 53310 (java) of user 988 killed by SIGABRT - dumping core
    elasticsearch.service: main process exited, code=killed, status=6/ABRT
    Failed to start Elasticsearch.

hs_err_pid log:

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f0475081b85, pid=44094, tid=44179

JRE version: OpenJDK Runtime Environment (13.0.2+8) (build 13.0.2+8)
Java VM: OpenJDK 64-Bit Server VM (13.0.2+8, mixed mode, sharing, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
Problematic frame:
C  [jna8520079277885295866.tmp+0x12b85]  ffi_prep_closure_loc+0x15

Core dump will be written. Default location: Core dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h" (or dumping to /usr/share/elasticsearch/core.44094)

If you would like to submit a bug report, please visit:
https://github.com/AdoptOpenJDK/openjdk-support/issues
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.


---------------  S U M M A R Y ------------

Command Line: -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=COMPAT -Xms8g -Xmx8g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/data1/els_dc1_1/tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=4294967296 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet

Host: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz, 4 cores, 15G, Red Hat Enterprise Linux Server release 7.7 (Maipo)

Extra Info:

---------------  T H R E A D  ---------------

Current thread (0x00007f04b8019000):  JavaThread "main" [_thread_in_native, id=44179, stack(0x00007f04c24e8000,0x00007f04c25e9000)]

Stack: [0x00007f04c24e8000,0x00007f04c25e9000],  sp=0x00007f04c25e6350,  free space=1016k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [jna8520079277885295866.tmp+0x12b85]  ffi_prep_closure_loc+0x15
C  [jna8520079277885295866.tmp+0x9ebc]  Java_com_sun_jna_Native_registerMethod+0x51c
j  com.sun.jna.Native.registerMethod(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;[I[J[JIJJLjava/lang/reflect/Method;JIZ[Lcom/sun/jna/ToNativeConverter;Lcom/sun/jna/FromNativeConverter;Ljava/lang/String;)J+0
j  com.sun.jna.Native.register(Ljava/lang/Class;Lcom/sun/jna/NativeLibrary;)V+1117
j  com.sun.jna.Native.register(Ljava/lang/Class;Ljava/lang/String;)V+17
j  com.sun.jna.Native.register(Ljava/lang/String;)V+7
j  org.elasticsearch.bootstrap.JNACLibrary.<clinit>()V+73
v  ~StubRoutines::call_stub
V  [libjvm.so+0x89e2d9]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x3c9
V  [libjvm.so+0x8710be]  InstanceKlass::call_class_initializer(Thread*)+0x22e
V  [libjvm.so+0x8716e4]  InstanceKlass::initialize_impl(Thread*)+0x4d4
V  [libjvm.so+0xacdb3f]  LinkResolver::resolve_static_call(CallInfo&, LinkInfo const&, bool, Thread*)+0x64f
V  [libjvm.so+0xacf839]  LinkResolver::resolve_invoke(CallInfo&, Handle, constantPoolHandle const&, int, Bytecodes::Code, Thread*)+0x2c9
V  [libjvm.so+0x8945ce]  InterpreterRuntime::resolve_invoke(JavaThread*, Bytecodes::Code)+0x2de
V  [libjvm.so+0x8953e5]  InterpreterRuntime::resolve_from_cache(JavaThread*, Bytecodes::Code)+0x115
j  org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot()Z+8
j  org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot()Z+18
j  org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Ljava/nio/file/Path;ZZZ)V+7
j  org.elasticsearch.bootstrap.Bootstrap.setup(ZLorg/elasticsearch/env/Environment;)V+71
j  org.elasticsearch.bootstrap.Bootstrap.init(ZLjava/nio/file/Path;ZLorg/elasticsearch/env/Environment;)V+234
j  org.elasticsearch.bootstrap.Elasticsearch.init(ZLjava/nio/file/Path;ZLorg/elasticsearch/env/Environment;)V+13
j  org.elasticsearch.bootstrap.Elasticsearch.execute(Lorg/elasticsearch/cli/Terminal;Ljoptsimple/OptionSet;Lorg/elasticsearch/env/Environment;)V+204
j  org.elasticsearch.cli.EnvironmentAwareCommand.execute(Lorg/elasticsearch/cli/Terminal;Ljoptsimple/OptionSet;)V+218
j  org.elasticsearch.cli.Command.mainWithoutErrorHandling([Ljava/lang/String;Lorg/elasticsearch/cli/Terminal;)V+79
j  org.elasticsearch.cli.Command.main([Ljava/lang/String;Lorg/elasticsearch/cli/Terminal;)I+47
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;Lorg/elasticsearch/bootstrap/Elasticsearch;Lorg/elasticsearch/cli/Terminal;)I+3
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+29
v  ~StubRoutines::call_stub
V  [libjvm.so+0x89e2d9]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x3c9
V  [libjvm.so+0x91b69a]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.71] [clone .constprop.228]+0x2fa
V  [libjvm.so+0x91dfdd]  jni_CallStaticVoidMethod+0x15d
C  [libjli.so+0x587d]  JavaMain+0xe4d
C  [libjli.so+0x9249]  ThreadJavaMain+0x9

Log:

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.sun.jna.Native.registerMethod(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/String;[I[J[JIJJLjava/lang/reflect/Method;JIZ[Lcom/sun/jna/ToNativeConverter;Lcom/sun/jna/FromNativeConverter;Ljava/lang/String;)J+0
j  com.sun.jna.Native.register(Ljava/lang/Class;Lcom/sun/jna/NativeLibrary;)V+1117
j  com.sun.jna.Native.register(Ljava/lang/Class;Ljava/lang/String;)V+17
j  com.sun.jna.Native.register(Ljava/lang/String;)V+7
j  org.elasticsearch.bootstrap.JNACLibrary.<clinit>()V+73
v  ~StubRoutines::call_stub
j  org.elasticsearch.bootstrap.JNANatives.definitelyRunningAsRoot()Z+8
j  org.elasticsearch.bootstrap.Natives.definitelyRunningAsRoot()Z+18
j  org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Ljava/nio/file/Path;ZZZ)V+7
j  org.elasticsearch.bootstrap.Bootstrap.setup(ZLorg/elasticsearch/env/Environment;)V+71
j  org.elasticsearch.bootstrap.Bootstrap.init(ZLjava/nio/file/Path;ZLorg/elasticsearch/env/Environment;)V+234
j  org.elasticsearch.bootstrap.Elasticsearch.init(ZLjava/nio/file/Path;ZLorg/elasticsearch/env/Environment;)V+13
j  org.elasticsearch.bootstrap.Elasticsearch.execute(Lorg/elasticsearch/cli/Terminal;Ljoptsimple/OptionSet;Lorg/elasticsearch/env/Environment;)V+204
j  org.elasticsearch.cli.EnvironmentAwareCommand.execute(Lorg/elasticsearch/cli/Terminal;Ljoptsimple/OptionSet;)V+218
j  org.elasticsearch.cli.Command.mainWithoutErrorHandling([Ljava/lang/String;Lorg/elasticsearch/cli/Terminal;)V+79
j  org.elasticsearch.cli.Command.main([Ljava/lang/String;Lorg/elasticsearch/cli/Terminal;)I+47
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;Lorg/elasticsearch/bootstrap/Elasticsearch;Lorg/elasticsearch/cli/Terminal;)I+3
j  org.elasticsearch.bootstrap.Elasticsearch.main([Ljava/lang/String;)V+29
v  ~StubRoutines::call_stub

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Register to memory mapping:

RAX=0x00007f0475082186: ffi_closure_unix64+0x0000000000000000 in /data1/els_dc1_1/tmp/jna--1985354563/jna8520079277885295866.tmp at 0x00007f047506f000
RBX=0x00007f04b8019310 points into unknown readable memory: 80 63 71 c1 04 7f 00 00
RCX=0x00007f04b880d050 points into unknown readable memory: 02 00 00 00 01 00 00 00
RDX=0x00007f047507b970: <offset 0x000000000000c970> in /data1/els_dc1_1/tmp/jna--1985354563/jna8520079277885295866.tmp at 0x00007f047506f000
RSP=0x00007f04c25e6350 is pointing into the stack for thread: 0x00007f04b8019000
RBP=0x00007f04c25e6350 is pointing into the stack for thread: 0x00007f04b8019000
RSI=0x00007f04b880d070 points into unknown readable memory: 02 00 00 00 03 00 00 00
RDI=0x0 is NULL
R8 =0x00007f04b8019000 is a thread
R9 =0x0 is NULL
R10=0x00007f04c25e5da0 is pointing into the stack for thread: 0x00007f04b8019000
R11=0x00007f04c1db0da0: __errno_location+0x0000000000000000 in /lib64/libpthread.so.0 at 0x00007f04c1da0000
R12=0x00007f04b880d050 points into unknown readable memory: 02 00 00 00 01 00 00 00
R13=0x00007f04b880d070 points into unknown readable memory: 02 00 00 00 03 00 00 00
R14=0x0 is NULL
R15=0x0000000000000004 is an unknown value


Registers:
RAX=0x00007f0475082186, RBX=0x00007f04b8019310, RCX=0x00007f04b880d050, RDX=0x00007f047507b970
RSP=0x00007f04c25e6350, RBP=0x00007f04c25e6350, RSI=0x00007f04b880d070, RDI=0x0000000000000000
R8 =0x00007f04b8019000, R9 =0x0000000000000000, R10=0x00007f04c25e5da0, R11=0x00007f04c1db0da0
R12=0x00007f04b880d050, R13=0x00007f04b880d070, R14=0x0000000000000000, R15=0x0000000000000004
RIP=0x00007f0475081b85, EFLAGS=0x0000000000010293, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f04c25e6350)
0x00007f04c25e6350:   00007f04c25e6810 00007f0475078ebc
0x00007f04c25e6360:   00007f04c25e6900 00007f04c25e6560
0x00007f04c25e6370:   00007f04b880d3e0 00007f04b880d3c0
0x00007f04c25e6380:   00007f04c25e69b0 0000000000000001 

Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007f04c25e6810 is pointing into the stack for thread: 0x00007f04b8019000
stack at sp + 1 slots: 0x00007f0475078ebc: Java_com_sun_jna_Native_registerMethod+0x000000000000051c in /data1/els_dc1_1/tmp/jna--1985354563/jna8520079277885295866.tmp at 0x00007f047506f000
stack at sp + 2 slots: 0x00007f04c25e6900 is pointing into the stack for thread: 0x00007f04b8019000
stack at sp + 3 slots: 0x00007f04c25e6560 is pointing into the stack for thread: 0x00007f04b8019000
stack at sp + 4 slots: 0x00007f04b880d3e0 points into unknown readable memory: 2f 2e 2e 2f 6c 69 62 2f
stack at sp + 5 slots: 0x00007f04b880d3c0 points into unknown readable memory: 2f 75 73 72 2f 73 68 61
stack at sp + 6 slots: 0x00007f04c25e69b0 is pointing into the stack for thread: 0x00007f04b8019000
stack at sp + 7 slots: 0x0000000000000001 is an unknown value

warkolm · May 4, 2020, 9:45pm

Please don't post pictures of text, they are difficult to read, impossible to search and some people may not be even able to see them

Black-Hat · May 4, 2020, 10:07pm

Amended as per advise, appreciate if the team can provide some informative guidance on this issue

warkolm · May 4, 2020, 10:13pm

Thanks. Thanks also for using code formatting to make things readable!

Can you post this file?

Also what does /var/log/elasticsearch/elasticsearch.log contain?

psramkumar · May 5, 2020, 3:19pm

What is the solution, i do not see any info relate to solution

pjanzen · May 5, 2020, 3:59pm

usually the solution lays within the elasticsearch log file. Mark Walkom is referring to that file and like to see the content of that file. Without it, it is very hard to determine the problem..

Black-Hat · May 5, 2020, 6:38pm

Hi, I tried default elasticsearch.yml without any configuration and also another attempts with setting below

cluster.name: elasticsearch
node.name: els-dc1-1
node.master: true
node.data: false
path.data: /data1/els_dc1_1/data
path.logs: /data1/els_dc1_1/log
network.host: 10.x.xxx.xxx

Planned to have a cluster of 3 consists of 1 Master node and 2 Data node [Will apply this setting for Data Node in future] Please correct me if I'm wrong:

cluster.name: elasticsearch
node.name: els-dc1-2
node.master: false
node.data: true
path.data: /data1/els_dc1_2/data
path.logs: /data1/els_dc1_2/log
network.host: 10.x.xxx.xxx
discovery.seed_hosts: ["10.xxxxx"]  << IP ADDRESS OF MASTER NODE

FYI, the host I install have the setting with noexec on /tmp, so I have modified the jvm.options as below:

## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms4g
-Xmx4g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

## JVM temporary directory
-Djava.io.tmpdir=/data1/els_dc1_1/tmp

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

Black-Hat · May 5, 2020, 6:43pm

Noticed that elasticsearach.log provide the info below:

[2020-05-05T15:55:48,241][WARN ][o.e.b.Natives            ] [xxxxxxxxxxxxxxx] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-7725357924971700307/jna--1985354563/jna2482199391639058917.tmp: /tmp/elasticsearch-7725357924971700307/jna--1985354563/jna2482199391639058917.tmp: failed to map segment from shared object: Operation not permitted
	at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
	at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
	at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
	at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
	at java.lang.System.load(System.java:1873) ~[?:?]
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
	at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
	at java.lang.Class.forName0(Native Method) ~[?:?]
	at java.lang.Class.forName(Class.java:333) ~[?:?]
	at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.6.2.jar:7.6.2]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.6.2.jar:7.6.2]
[2020-05-05T15:55:48,249][WARN ][o.e.b.Natives            ] [xxxxxxxxxxxxxxx] cannot check if running as root because JNA is not available
[2020-05-05T15:55:48,249][WARN ][o.e.b.Natives            ] [xxxxxxxxxxxxxxx] cannot install system call filter because JNA is not available
[2020-05-05T15:55:48,250][WARN ][o.e.b.Natives            ] [xxxxxxxxxxxxxxx] cannot register console handler because JNA is not available
[2020-05-05T15:55:48,251][WARN ][o.e.b.Natives            ] [xxxxxxxxxxxxxxx] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2020-05-05T15:55:48,251][WARN ][o.e.b.Natives            ] [xxxxxxxxxxxxxxx] cannot getrlimit RLIMIT_AS because JNA is not available
[2020-05-05T15:55:48,252][WARN ][o.e.b.Natives            ] [xxxxxxxxxxxxxxx] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2020-05-05T15:55:48,419][INFO ][o.e.e.NodeEnvironment    ] [xxxxxxxxxxxxxxx] using [1] data paths, mounts [[/var (/dev/mapper/osvg-var)]], net usable_space [3gb], net total_space [4.6gb], types [xfs]
[2020-05-05T15:55:48,420][INFO ][o.e.e.NodeEnvironment    ] [xxxxxxxxxxxxxxx] heap size [990.7mb], compressed ordinary object pointers [true]
[2020-05-05T15:55:48,463][INFO ][o.e.n.Node               ] [xxxxxxxxxxxxxxx] node name [xxxxxxxxxxxxxxx], node ID [racaeHmzTumhXPtCsFK0QQ], cluster name [elasticsearch]
[2020-05-05T15:55:48,464][INFO ][o.e.n.Node               ] [xxxxxxxxxxxxxxx] version[7.6.2], pid[93865], build[default/rpm/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/3.10.0-1062.18.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.2/13.0.2+8]
[2020-05-05T15:55:48,464][INFO ][o.e.n.Node               ] [xxxxxxxxxxxxxxx] JVM home [/usr/share/elasticsearch/jdk]
[2020-05-05T15:55:48,464][INFO ][o.e.n.Node               ] [xxxxxxxxxxxxxxx] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-7725357924971700307, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
.
.
.
.
[2020-05-05T15:55:53,606][INFO ][o.e.x.s.a.s.FileRolesStore] [xxxxxxxxxxxxxxx] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-05-05T15:55:54,050][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [xxxxxxxxxxxxxxx] [controller/93967] [Main.cc@110] controller (64 bit): Version 7.6.2 (Build e06ef9d86d5332) Copyright (c) 2020 Elasticsearch BV
[2020-05-05T15:55:54,568][DEBUG][o.e.a.ActionModule       ] [xxxxxxxxxxxxxxx] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2020-05-05T15:55:54,695][INFO ][o.e.d.DiscoveryModule    ] [xxxxxxxxxxxxxxx] using discovery type [zen] and seed hosts providers [settings]
[2020-05-05T15:55:55,570][INFO ][o.e.n.Node               ] [xxxxxxxxxxxxxxx] initialized
[2020-05-05T15:55:55,571][INFO ][o.e.n.Node               ] [xxxxxxxxxxxxxxx] starting ...
[2020-05-05T15:55:55,704][INFO ][o.e.t.TransportService   ] [xxxxxxxxxxxxxxx] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2020-05-05T15:55:55,935][WARN ][o.e.b.BootstrapChecks    ] [xxxxxxxxxxxxxxx] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2020-05-05T15:55:55,935][WARN ][o.e.b.BootstrapChecks    ] [xxxxxxxxxxxxxxx] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-05-05T15:55:55,945][INFO ][o.e.c.c.ClusterBootstrapService] [xxxxxxxxxxxxxxx] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-05-05T15:55:58,952][INFO ][o.e.c.c.Coordinator      ] [xxxxxxxxxxxxxxx] setting initial configuration to VotingConfiguration{racaeHmzTumhXPtCsFK0QQ}
[2020-05-05T15:55:59,203][INFO ][o.e.c.s.MasterService    ] [xxxxxxxxxxxxxxx] elected-as-master ([1] nodes joined)[{xxxxxxxxxxxxxxx}{racaeHmzTumhXPtCsFK0QQ}{p5E3EM_OSwOJhgY7RUqUGA}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=16806420480, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: master node changed {previous [], current [{xxxxxxxxxxxxxxx}{racaeHmzTumhXPtCsFK0QQ}{p5E3EM_OSwOJhgY7RUqUGA}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=16806420480, xpack.installed=true, ml.max_open_jobs=20}]}
[2020-05-05T15:55:59,241][INFO ][o.e.c.c.CoordinationState] [xxxxxxxxxxxxxxx] cluster UUID set to [9fzd7MjoRDyJkTAEilDotQ]
[2020-05-05T15:55:59,263][INFO ][o.e.c.s.ClusterApplierService] [xxxxxxxxxxxxxxx] master node changed {previous [], current [{xxxxxxxxxxxxxxx}{racaeHmzTumhXPtCsFK0QQ}{p5E3EM_OSwOJhgY7RUqUGA}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=16806420480, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{term=1, version=1}
[2020-05-05T15:55:59,307][INFO ][o.e.h.AbstractHttpServerTransport] [xxxxxxxxxxxxxxx] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2020-05-05T15:55:59,307][INFO ][o.e.n.Node               ] [xxxxxxxxxxxxxxx] started
[2020-05-05T15:55:59,308][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxxxxxxxxxxxxxx] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
	at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
	at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
	at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:114) ~[?:?]
	at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:124) ~[?:?]
	at java.util.ArrayList.forEach(ArrayList.java:1507) ~[?:?]
	at org.elasticsearch.node.Node.start(Node.java:787) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:273) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.2.jar:7.6.2]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.2.jar:7.6.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.2.jar:7.6.2]

I've pointed the temp java directory as below:

-Djava.io.tmpdir=/data1/els_dc1_1/tmp

For the mount on RHEL:

/dev/mapper/vgdb-data1 on /data1 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

For /etc/fstab/ on RHEL:

/dev/mapper/vgdb-data1  /data1 xfs      defaults        0 0

For the custom tmp directory /data1/els_dc1_1/tmp on RHEL:

drwxrwxrwx. 3 appadmin elasticsearch 29 May  6 02:29 tmp

Luca_Belluccini · May 5, 2020, 7:14pm

This might be related to https://github.com/elastic/elasticsearch/pull/44673 and in particular https://www.elastic.co/guide/en/elasticsearch/reference/current/executable-jna-tmpdir.html

I am on my phone so I cannot dig further.

Black-Hat · May 5, 2020, 7:21pm

Hi, my /data1 directory have no limitation on the noexec attribute.

/data1/els_dc1_1/tmp

Also, I've another application pointed under this directory as tmp location for JRE and it's working fine

/data1/sample/tmp

Ivan_A_Carrazana_C · May 6, 2020, 8:24pm

Hi
If you are applying a security compliance in your RHEL installation you must change the path of the TMP directory that will use elasticsearch as Java.

Uncomment at /etc/elasticsearch/jvm.options
-Djava.io.tmpdir=${ES_TMPDIR}

Add in /etc /sysconfig/elasticsearch
ES_TMPDIR=/usr/share/elasticsearch/tmp

Create the /usr/share/elasticsearch/tmp directory and make sure that the owner and group are elasticsearch and the permissions are 0755

Lastly make sure that /dev/shm doesn't have the noexec attribute with command:
mount | grep tmpfs | grep '/dev/shm'
Expected result:
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)

If you get output like these:
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,seclabel)
Add or modify in /etc/fstab the following line:
tmpfs /dev/shm tmpfs defaults,nodev,nosuid 0 0

I had the same problem and this worked for me. Hope i can help you

Black-Hat · May 7, 2020, 10:35am

Hi Ivan,
Thanks for your advise, will try it on later.

But why is this related to /dev/shm folder permission? Since we have mapped the Djava.io.tmpdir to another directory.

Let's say in the approach given by you:

ES_TMPDIR=/usr/share/elasticsearch/tmp

Shouldn't we ensure /usr/share folder have no attribute of noexec? What's the relationship with /usr/shm?

Appreciate if you can help to further explain this.

Thanks

Black-Hat · May 7, 2020, 6:48pm

Hi Ivan, thanks for your advice. Finally I managed to setup entire Elasticsearch cluster.

Seriously, Thanks a lot!!!!!

system · June 4, 2020, 6:48pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.