Elasticsearch not starting after RHEL kernel upgrade

Hi,

Elasticsearch (v7.5.2) does not start after performing a kernel upgrade from 3.10.0-957.el7 to 3.10.0-1160.80.1.el7.

[2022-12-28T12:30:27,741][INFO ][o.e.b.BootstrapChecks    ] [igw-edb3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-12-28T12:30:27,776][ERROR][o.e.b.Bootstrap          ] [igw-edb3] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2022-12-28T12:30:27,780][INFO ][o.e.n.Node               ] [igw-edb3] stopping ...
[2022-12-28T12:30:27,795][INFO ][o.e.n.Node               ] [igw-edb3] stopped
[2022-12-28T12:30:27,795][INFO ][o.e.n.Node               ] [igw-edb3] closing ...
[2022-12-28T12:30:27,811][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [igw-edb3] fatal error in thread [Thread-3], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:114) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.close(SystemdPlugin.java:139) ~[?:?]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:86) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.node.Node.close(Node.java:876) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:62) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:204) ~[elasticsearch-7.5.2.jar:7.5.2]

Java version is as below.

openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13.0.1+9, mixed mode, sharing)

This is most probably caused by the kernel upgrade, could you please help us resolve this issue?

Welcome to our community! :smiley:

Elasticsearch 7.5 is EOL and no longer supported, you will need to upgrade.

Hi @warkolm , thanks for the info.

However, until the cluster is upgraded, could you please share any insight as to why the issue is occurring with regards to ES internals, and a possible workaround if available for the time being?

7.5 is no longer supported and the only path forward would be to upgrade sorry.

Are you sure there aren't additional lines in that error message?

Hi @TimV , above was truncated for brevity, please find the complete log below.

[2022-12-28T12:30:17,862][WARN ][o.e.b.Natives            ] [igw-edb3] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-11796208705730343141/jna--1985354563/jna6225614801317523701.tmp: /tmp/elasticsearch-11796208705730343141/jna--1985354563/jna6225614801317523701.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]
        at java.lang.System.load(System.java:1873) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:333) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:190) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:369) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.5.2.jar:7.5.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.5.2.jar:7.5.2]
[2022-12-28T12:30:17,872][WARN ][o.e.b.Natives            ] [igw-edb3] cannot check if running as root because JNA is not available
[2022-12-28T12:30:17,872][WARN ][o.e.b.Natives            ] [igw-edb3] cannot install system call filter because JNA is not available
[2022-12-28T12:30:17,873][WARN ][o.e.b.Natives            ] [igw-edb3] cannot register console handler because JNA is not available
[2022-12-28T12:30:17,874][WARN ][o.e.b.Natives            ] [igw-edb3] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2022-12-28T12:30:17,875][WARN ][o.e.b.Natives            ] [igw-edb3] cannot getrlimit RLIMIT_AS because JNA is not available
[2022-12-28T12:30:17,875][WARN ][o.e.b.Natives            ] [igw-edb3] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2022-12-28T12:30:18,103][INFO ][o.e.e.NodeEnvironment    ] [igw-edb3] using [1] data paths, mounts [[/var/log (/dev/mapper/rhel-var_log)]], net usable_space [138.5gb], net total_space [492gb], types [ext4]
[2022-12-28T12:30:18,105][INFO ][o.e.e.NodeEnvironment    ] [igw-edb3] heap size [7.9gb], compressed ordinary object pointers [true]
[2022-12-28T12:30:18,233][INFO ][o.e.n.Node               ] [igw-edb3] node name [igw-edb3], node ID [E7UHwZDOSvWHwWAf7iEQ7w], cluster name [production]
[2022-12-28T12:30:18,233][INFO ][o.e.n.Node               ] [igw-edb3] version[7.5.2], pid[13050], build[default/rpm/8bec50e1e0ad29dad5653712cf3bb580cd1afcdf/2020-01-15T12:11:52.313576Z], OS[Linux/3.10.0-1160.80.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]
[2022-12-28T12:30:18,234][INFO ][o.e.n.Node               ] [igw-edb3] JVM home [/usr/share/elasticsearch/jdk]
[2022-12-28T12:30:18,234][INFO ][o.e.n.Node               ] [igw-edb3] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms8g, -Xmx8g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-11796208705730343141, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -XX:MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2022-12-28T12:30:20,200][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [aggs-matrix-stats]
[2022-12-28T12:30:20,201][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [analysis-common]
[2022-12-28T12:30:20,201][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [flattened]
[2022-12-28T12:30:20,201][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [frozen-indices]
[2022-12-28T12:30:20,201][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [ingest-common]
[2022-12-28T12:30:20,202][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [ingest-geoip]
[2022-12-28T12:30:20,202][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [ingest-user-agent]
[2022-12-28T12:30:20,202][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [lang-expression]
[2022-12-28T12:30:20,202][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [lang-mustache]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [lang-painless]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [mapper-extras]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [parent-join]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [percolator]
[2022-12-28T12:30:20,203][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [rank-eval]
[2022-12-28T12:30:20,204][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [reindex]
[2022-12-28T12:30:20,204][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [repository-url]
[2022-12-28T12:30:20,204][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [search-business-rules]
[2022-12-28T12:30:20,204][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [spatial]
[2022-12-28T12:30:20,205][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [systemd]
[2022-12-28T12:30:20,205][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [transform]
[2022-12-28T12:30:20,205][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [transport-netty4]
[2022-12-28T12:30:20,205][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [vectors]
[2022-12-28T12:30:20,206][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-analytics]
[2022-12-28T12:30:20,206][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-ccr]
[2022-12-28T12:30:20,206][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-core]
[2022-12-28T12:30:20,206][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-deprecation]
[2022-12-28T12:30:20,207][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-enrich]
[2022-12-28T12:30:20,207][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-graph]
[2022-12-28T12:30:20,207][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-ilm]
[2022-12-28T12:30:20,207][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-logstash]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-ml]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-monitoring]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-rollup]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-security]
[2022-12-28T12:30:20,208][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-sql]
[2022-12-28T12:30:20,209][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-voting-only-node]
[2022-12-28T12:30:20,209][INFO ][o.e.p.PluginsService     ] [igw-edb3] loaded module [x-pack-watcher]
[2022-12-28T12:30:20,209][INFO ][o.e.p.PluginsService     ] [igw-edb3] no plugins loaded
[2022-12-28T12:30:24,554][INFO ][o.e.x.s.a.s.FileRolesStore] [igw-edb3] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-12-28T12:30:25,183][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [igw-edb3] [controller/13154] [Main.cc@110] controller (64 bit): Version 7.5.2 (Build 68f6981dfb8e2d) Copyright (c) 2020 Elasticsearch BV
[2022-12-28T12:30:25,883][DEBUG][o.e.a.ActionModule       ] [igw-edb3] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2022-12-28T12:30:26,024][INFO ][o.e.d.DiscoveryModule    ] [igw-edb3] using discovery type [zen] and seed hosts providers [settings]
[2022-12-28T12:30:27,019][INFO ][o.e.n.Node               ] [igw-edb3] initialized
[2022-12-28T12:30:27,020][INFO ][o.e.n.Node               ] [igw-edb3] starting ...
[2022-12-28T12:30:27,193][INFO ][o.e.t.TransportService   ] [igw-edb3] publish_address {10.74.2.168:9300}, bound_addresses {[::]:9300}
[2022-12-28T12:30:27,741][INFO ][o.e.b.BootstrapChecks    ] [igw-edb3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-12-28T12:30:27,776][ERROR][o.e.b.Bootstrap          ] [igw-edb3] node validation exception
[1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2022-12-28T12:30:27,780][INFO ][o.e.n.Node               ] [igw-edb3] stopping ...
[2022-12-28T12:30:27,795][INFO ][o.e.n.Node               ] [igw-edb3] stopped
[2022-12-28T12:30:27,795][INFO ][o.e.n.Node               ] [igw-edb3] closing ...
[2022-12-28T12:30:27,811][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [igw-edb3] fatal error in thread [Thread-3], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:114) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.close(SystemdPlugin.java:139) ~[?:?]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:86) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.node.Node.close(Node.java:876) ~[elasticsearch-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:104) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:62) ~[elasticsearch-core-7.5.2.jar:7.5.2]
        at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:204) ~[elasticsearch-7.5.2.jar:7.5.2]

Much appreciate the support, thanks.

Try this: JNA temporary directory not mounted with noexec | Elasticsearch Guide [7.5] | Elastic

Hi @TimV ,

That was helpful, thanks again!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.