Elasticsearch service not starting

gsumilhig · January 10, 2020, 5:42am

Caused by: java.lang.IllegalStateException: jar hell!
class: org.elasticsearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/elasticsearch/lib/plugin-classloader-6.8.0.jar
jar2: /usr/share/elasticsearch/lib/plugin-classloader-6.8.3.jar
at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:277) ~[elasticsearch-core-6.8.3.jar:6.8.3]
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:190) ~[elasticsearch-core-6.8.3.jar:6.8.3]
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86) ~[elasticsearch-core-6.8.3.jar:6.8.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:197) ~[elasticsearch-6.8.3.jar:6.8.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.3.jar:6.8.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.3.jar:6.8.3]
... 6 more

dadoonet · January 10, 2020, 5:59am

Welcome!

It could help if you explain what you did exactly to get in that situation.
Here it seems that 2 versions are mixed together.

gsumilhig · January 10, 2020, 6:16am

Hi,

We just shutdown the VM and increase the disk because /var/log/elasticsearh is 100% full.. after that we found that ther service didn't start.

systemctl status elasticsearch

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2020-01-10 13:51:50 PST; 24min ago
Docs: http://www.elastic.co
Process: 4637 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 4637 (code=exited, status=1/FAILURE)

Jan 10 13:51:47 elastic01 systemd[1]: Started Elasticsearch.
Jan 10 13:51:50 elastic01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jan 10 13:51:50 elastic01 systemd[1]: Unit elasticsearch.service entered failed state.
Jan 10 13:51:50 elastic01 systemd[1]: elasticsearch.service failed.

gsumilhig · January 10, 2020, 6:19am

We have 3 nodes and we applied the disk increase in the first node only.

dadoonet · January 10, 2020, 6:35am

Pretty sure that something else happened.
Like an upgrade which was wrongly executed.

gsumilhig · January 10, 2020, 7:06am

Hi,

Is it possible to delete some files in cluster /var/log/elasticsearch?

dadoonet · January 10, 2020, 7:37am

The trouble seems to be in the plugins dir here.
What do you have inside?

gsumilhig · January 10, 2020, 7:41am

there's no file inside /usr/share/elasticsearch/plugins. same with the other nodes

dadoonet · January 10, 2020, 7:59am

Could you share the full elasticsearch logs please?

Please format your logs. Paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

gsumilhig · January 10, 2020, 9:14am

java.lang.IllegalStateException: jar hell!
class: org.elasticsearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/elasticsearch/lib/plugin-classloader-6.8.0.jar
jar2: /usr/share/elasticsearch/lib/plugin-classloader-6.8.3.jar
        at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:277) ~[elasticsearch-core-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:190) ~[elasticsearch-core-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86) ~[elasticsearch-core-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:197) ~[elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.0.jar:6.8.3]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.0.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.3.jar:6.8.3]
[2020-01-10T14:52:27,833][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elastic01] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.elasticsearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/elasticsearch/lib/plugin-classloader-6.8.0.jar
jar2: /usr/share/elasticsearch/lib/plugin-classloader-6.8.3.jar
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.0.jar:6.8.3]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.0.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.3.jar:6.8.3]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.elasticsearch.plugins.ExtendedPluginsClassLoader
jar1: /usr/share/elasticsearch/lib/plugin-classloader-6.8.0.jar
jar2: /usr/share/elasticsearch/lib/plugin-classloader-6.8.3.jar
        at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:277) ~[elasticsearch-core-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:190) ~[elasticsearch-core-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86) ~[elasticsearch-core-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:197) ~[elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.3.jar:6.8.3]
        ... 6 more```

Thank you.

dadoonet · January 10, 2020, 9:57am

I see.

Could list the files which are in the elasticsearch lib directory?

gsumilhig · January 13, 2020, 1:46am

ls /usr/share/elasticsearch/lib/

elasticsearch-6.8.0.jar hppc-0.7.1.jar log4j-api-2.11.1.jar lucene-sandbox-7.7.0.jar
elasticsearch-6.8.3.jar jackson-core-2.8.11.jar log4j-core-2.11.1.jar lucene-spatial3d-7.7.0.jar
elasticsearch-cli-6.8.0.jar jackson-dataformat-cbor-2.8.11.jar lucene-analyzers-common-7.7.0.jar lucene-spatial-7.7.0.jar
elasticsearch-core-6.8.0.jar jackson-dataformat-smile-2.8.11.jar lucene-backward-codecs-7.7.0.jar lucene-spatial-extras-7.7.0.jar
elasticsearch-core-6.8.3.jar jackson-dataformat-yaml-2.8.11.jar lucene-core-7.7.0.jar lucene-suggest-7.7.0.jar
elasticsearch-launchers-6.8.0.jar java-version-checker-6.8.0.jar lucene-grouping-7.7.0.jar plugin-classloader-6.8.0.jar
elasticsearch-launchers-6.8.3.jar java-version-checker-6.8.3.jar lucene-highlighter-7.7.0.jar plugin-classloader-6.8.3.jar
elasticsearch-secure-sm-6.8.0.jar jna-4.5.1.jar lucene-join-7.7.0.jar snakeyaml-1.17.jar
elasticsearch-secure-sm-6.8.3.jar joda-time-2.10.1.jar lucene-memory-7.7.0.jar spatial4j-0.7.jar
elasticsearch-x-content-6.8.0.jar jopt-simple-5.0.2.jar lucene-misc-7.7.0.jar t-digest-3.2.jar
elasticsearch-x-content-6.8.3.jar jts-core-1.15.0.jar lucene-queries-7.7.0.jar tools
HdrHistogram-2.1.9.jar log4j-1.2-api-2.11.1.jar lucene-queryparser-7.7.0.jar

dadoonet · January 13, 2020, 5:05am

You have a mix of 2 elasticsearch versions.
You should reinstall correctly elasticsearch but keep the data dir and the config dir.

gsumilhig · January 13, 2020, 6:02am

Error after reinstall elasticsearch using same version.

[2020-01-13T13:38:50,082][WARN ][o.e.b.Natives            ] [elastic01] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-8296603538933134483/jna--1985354563/jna6449887839057156505.tmp: /tmp/elasticsearch-8296603538933134483/jna--1985354563/jna6449887839057156505.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[?:1.8.0_222]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[?:1.8.0_222]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[?:1.8.0_222]
        at java.lang.Runtime.load0(Runtime.java:809) ~[?:1.8.0_222]
        at java.lang.System.load(System.java:1086) ~[?:1.8.0_222]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_222]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_222]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:102) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.0.jar:6.8.3]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.0.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.3.jar:6.8.3]
[2020-01-13T13:38:50,089][WARN ][o.e.b.Natives            ] [elastic01] cannot check if running as root because JNA is not available
[2020-01-13T13:38:50,089][WARN ][o.e.b.Natives            ] [elastic01] cannot register console handler because JNA is not available
[2020-01-13T13:38:50,090][WARN ][o.e.b.Natives            ] [elastic01] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2020-01-13T13:38:50,090][WARN ][o.e.b.Natives            ] [elastic01] cannot getrlimit RLIMIT_AS because JNA is not available
[2020-01-13T13:38:50,091][WARN ][o.e.b.Natives            ] [elastic01] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2020-01-13T13:38:50,125][ERROR][o.e.b.Bootstrap          ] [elastic01] Exception
java.lang.IllegalStateException: jar hell!
class: META-INF.versions.9.org.elasticsearch.monitor.jvm.JvmPid
jar1: /usr/share/elasticsearch/lib/elasticsearch-6.8.3.jar
jar2: /usr/share/elasticsearch/lib/elasticsearch-6.8.0.jar
        at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:277) ~[elasticsearch-core-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:190) ~[elasticsearch-core-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:86) ~[elasticsearch-core-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:197) ~[elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.0.jar:6.8.3]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.0.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.3.jar:6.8.3]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.3.jar:6.8.3]```

gsumilhig · January 13, 2020, 7:43am

it works now, by uninstalling elasticsearch and remove some elasticsearch file lib..

Thank you dadoonet

system · February 10, 2020, 7:43am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.