Elasticsearch not starting after RHEL kernel upgrade

hasith-adl · December 28, 2022, 8:01am

Hi,

Elasticsearch (v7.5.2) does not start after performing a kernel upgrade from 3.10.0-957.el7 to 3.10.0-1160.80.1.el7.

[2022-12-28T12:30:27,741][INFO ][o.e.b.BootstrapChecks    ] [igw-edb3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-12-28T12:30:27,776][ERROR][o.e.b.Bootstrap          ] [igw-edb3] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2022-12-28T12:30:27,780][INFO ][o.e.n.Node               ] [igw-edb3] stopping ...
[2022-12-28T12:30:27,795][INFO ][o.e.n.Node               ] [igw-edb3] stopped
[2022-12-28T12:30:27,795][INFO ][o.e.n.Node               ] [igw-edb3] closing ...
[2022-12-28T12:30:27,811][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [igw-edb3] fatal error in thread [Thread-3], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:114) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.close(SystemdPlugin.java:139) ~[?:?]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:86) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.node.Node.close(Node.java:876) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:62) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:204) ~[elasticsearch-7.5.2.jar:7.5.2]

Java version is as below.

openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13.0.1+9, mixed mode, sharing)

This is most probably caused by the kernel upgrade, could you please help us resolve this issue?

warkolm · December 30, 2022, 2:19am

Welcome to our community!

Elasticsearch 7.5 is EOL and no longer supported, you will need to upgrade.

hasith-adl · January 2, 2023, 6:19am

Hi @warkolm , thanks for the info.

However, until the cluster is upgraded, could you please share any insight as to why the issue is occurring with regards to ES internals, and a possible workaround if available for the time being?

warkolm · January 2, 2023, 9:12pm

7.5 is no longer supported and the only path forward would be to upgrade sorry.

TimV · January 3, 2023, 3:15am

Are you sure there aren't additional lines in that error message?

hasith-adl · January 4, 2023, 1:45am

Hi @TimV , above was truncated for brevity, please find the complete log below.

[2022-12-28T12:30:17,862][WARN ][o.e.b.Natives            ] [igw-edb3] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-11796208705730343141/jna--1985354563/jna6225614801317523701.tmp: /tmp/elasticsearch-11796208705730343141/jna--1985354563/jna6225614801317523701.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
        at java.lang.System.load(System.java:1873) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:333) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:190) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:369) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.5.2.jar:7.5.2]
[2022-12-28T12:30:17,872][WARN ][o.e.b.Natives            ] [igw-edb3] cannot check if running as root because JNA is not available
[2022-12-28T12:30:17,872][WARN ][o.e.b.Natives            ] [igw-edb3] cannot install system call filter because JNA is not available
[2022-12-28T12:30:17,873][WARN ][o.e.b.Natives            ] [igw-edb3] cannot register console handler because JNA is not available
[2022-12-28T12:30:17,874][WARN ][o.e.b.Natives            ] [igw-edb3] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2022-12-28T12:30:17,875][WARN ][o.e.b.Natives            ] [igw-edb3] cannot getrlimit RLIMIT_AS because JNA is not available
[2022-12-28T12:30:17,875][WARN ][o.e.b.Natives            ] [igw-edb3] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2022-12-28T12:30:18,103][INFO ][o.e.e.NodeEnvironment    ] [igw-edb3] using [1] data paths, mounts [[/var/log (/dev/mapper/rhel-var_log)]], net usable_space [138.5gb], net total_space [492gb], types [ext4]
[2022-12-28T12:30:18,105][INFO ][o.e.e.NodeEnvironment    ] [igw-edb3] heap size [7.9gb], compressed ordinary object pointers [true]
[2022-12-28T12:30:18,233][INFO ][o.e.n.Node               ] [igw-edb3] node name [igw-edb3], node ID [E7UHwZDOSvWHwWAf7iEQ7w], cluster name [production]
[2022-12-28T12:30:18,233][INFO ][o.e.n.Node               ] [igw-edb3] version[7.5.2], pid[13050], build[default/rpm/8bec50e1e0ad29dad5653712cf3bb580cd1afcdf/2020-01-15T12:11:52.313576Z], OS[Linux/3.10.0-1160.80.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]
[2022-12-28T12:30:18,234][INFO ][o.e.n.Node               ] [igw-edb3] JVM home [/usr/share/elasticsearch/jdk]
[2022-12-28T12:30:18,234][INFO ][o.e.n.Node               ] [igw-edb3] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms8g, -Xmx8g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-11796208705730343141, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -XX:MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2022-12-28T12:30:20,200][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [aggs-matrix-stats]
[2022-12-28T12:30:20,201][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [analysis-common]
[2022-12-28T12:30:20,201][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [flattened]
[2022-12-28T12:30:20,201][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [frozen-indices]
[2022-12-28T12:30:20,201][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [ingest-common]
[2022-12-28T12:30:20,202][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [ingest-geoip]
[2022-12-28T12:30:20,202][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [ingest-user-agent]
[2022-12-28T12:30:20,202][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [lang-expression]
[2022-12-28T12:30:20,202][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [lang-mustache]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [lang-painless]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [mapper-extras]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [parent-join]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [percolator]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [rank-eval]
[2022-12-28T12:30:20,204][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [reindex]
[2022-12-28T12:30:20,204][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [repository-url]
[2022-12-28T12:30:20,204][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [search-business-rules]
[2022-12-28T12:30:20,204][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [spatial]
[2022-12-28T12:30:20,205][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [systemd]
[2022-12-28T12:30:20,205][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [transform]
[2022-12-28T12:30:20,205][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [transport-netty4]
[2022-12-28T12:30:20,205][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [vectors]
[2022-12-28T12:30:20,206][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-analytics]
[2022-12-28T12:30:20,206][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-ccr]
[2022-12-28T12:30:20,206][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-core]
[2022-12-28T12:30:20,206][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-deprecation]
[2022-12-28T12:30:20,207][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-enrich]
[2022-12-28T12:30:20,207][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-graph]
[2022-12-28T12:30:20,207][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-ilm]
[2022-12-28T12:30:20,207][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-logstash]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-ml]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-monitoring]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-rollup]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-security]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-sql]
[2022-12-28T12:30:20,209][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-voting-only-node]
[2022-12-28T12:30:20,209][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-watcher]
[2022-12-28T12:30:20,209][INFO ][o.e.p.PluginsService     ] [igw-edb3] no plugins loaded
[2022-12-28T12:30:24,554][INFO ][o.e.x.s.a.s.FileRolesStore] [igw-edb3] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-12-28T12:30:25,183][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [igw-edb3] [controller/13154] [Main.cc@110] controller (64 bit): Version 7.5.2 (Build 68f6981dfb8e2d) Copyright (c) 2020 Elasticsearch BV
[2022-12-28T12:30:25,883][DEBUG][o.e.a.ActionModule       ] [igw-edb3] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2022-12-28T12:30:26,024][INFO ][o.e.d.DiscoveryModule    ] [igw-edb3] using discovery type [zen] and seed hosts providers [settings]
[2022-12-28T12:30:27,019][INFO ][o.e.n.Node               ] [igw-edb3] initialized
[2022-12-28T12:30:27,020][INFO ][o.e.n.Node               ] [igw-edb3] starting ...
[2022-12-28T12:30:27,193][INFO ][o.e.t.TransportService   ] [igw-edb3] publish_address {10.74.2.168:9300}, bound_addresses {[::]:9300}
[2022-12-28T12:30:27,741][INFO ][o.e.b.BootstrapChecks    ] [igw-edb3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-12-28T12:30:27,776][ERROR][o.e.b.Bootstrap          ] [igw-edb3] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2022-12-28T12:30:27,780][INFO ][o.e.n.Node               ] [igw-edb3] stopping ...
[2022-12-28T12:30:27,795][INFO ][o.e.n.Node               ] [igw-edb3] stopped
[2022-12-28T12:30:27,795][INFO ][o.e.n.Node               ] [igw-edb3] closing ...
[2022-12-28T12:30:27,811][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [igw-edb3] fatal error in thread [Thread-3], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:114) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.close(SystemdPlugin.java:139) ~[?:?]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:86) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.node.Node.close(Node.java:876) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:62) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:204) ~[elasticsearch-7.5.2.jar:7.5.2]

Much appreciate the support, thanks.

TimV · January 5, 2023, 5:10am

hasith-adl:

java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-11796208705730343141/jna--1985354563/jna6225614801317523701.tmp: /tmp/elasticsearch-11796208705730343141/jna--1985354563/jna6225614801317523701.tmp: failed to map segment from shared object: Operation not permitted

Try this: JNA temporary directory not mounted with noexec | Elasticsearch Guide [7.5] | Elastic

hasith-adl · January 12, 2023, 6:22am

Hi @TimV ,

That was helpful, thanks again!

system · February 9, 2023, 6:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.