Elastic search docker image - Jar hell error

I am using docker.elastic.co/elasticsearch/elasticsearch:5.6.16 as base image and trying to upgrade the jackson packages to resolve Critical CVE.

Dockerfile:

# https://github.com/elastic/elasticsearch-docker

FROM docker.elastic.co/elasticsearch/elasticsearch:5.6.16
# Remove existing jackson package
RUN rm -f /usr/share/elasticsearch/lib/jackson*


# # Installing patched Jackson packages
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.14.0/jackson-databind-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.14.0/jackson-core-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.14.0/jackson-annotations-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.14.0/jackson-dataformat-cbor-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.14.0/jackson-dataformat-yaml-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.14.0/jackson-dataformat-smile-2.14.0.jar -P /usr/share/elasticsearch/lib/

When this image is deployed as a pod and we tried to login, the pod went to crashloop with JAR Hell error

Error:

OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[2023-04-11T08:29:49,361][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: META-INF.versions.9.module-info
jar1: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.14.0.jar
jar2: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.14.0.jar
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.16.jar:5.6.16]
Caused by: java.lang.IllegalStateException: jar hell!
class: META-INF.versions.9.module-info
jar1: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.14.0.jar
jar2: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.14.0.jar
	at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:282) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:90) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.16.jar:5.6.16]
	... 6 more

Does elasticsearch of version 5.6.16 has explicit dependency on Jackson 2.8.16 package version or we can upgrade Jackson package to 2.14.0 without compatibility issues??

Please help!!!

Welcome!

You should really think of switching to a recent version instead.

2 Likes

Hi David, unfortunately we have a dependent on this particular version. We are planning to upgrade eventually. But this is an urgent requirement to patch Critical CVE. Is there any dependency with Jackson version 2.8.16 or we can upgrade the Jackson package version ?? Please help.

Thanks

It's really a matter of urgency that you upgrade your nodes.
Fixing that CVE won't fix all the security issues that you might have with this so old version.

But, back to your question. If I recall correctly how I was building Elasticsearch in the past, I'd probably:

  • checkout the code from GitHub
  • upgrade the version in the pom.xml file
  • compute again the signatures for all the jars (not sure if it's needed and how but the next step should tell you)
  • build the project

I don't think you can just replace a jar as is.

2 Likes

Thank you for the response.
Since we are using the base docker image docker.elastic.co/elasticsearch/elasticsearch:5.6.16 which is pre-built. Is there any way to upgrade the version on top of the pre-built code source?

I believe (hope) that the build does produce the image.

Hi David, I tried to follow the steps you suggested.

  1. Checked out github code for elasticsearch 5.6 version
  2. There was no pom.xml file in the source code. I could only find jackson version under in buildSrc/version.properties file and updated it.
  3. Ran -> ./gradlew localDistro to build from source code. The build failed.

Is this the correct way to upgrade jackson version? Can you please help with some docker instructions on how to upgrade this with custom dockerfile instead of updating the source code?

elasticsearch 5.6 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

1 Like

Ha! I did not remember when we exactly switched to Gradle.

What is the error message in the build?

22:32:07.944 [INFO] [org.gradle.internal.nativeintegration.services.NativeServices] Initialized native services in: /home/user/.gradle/native

22:32:08.059 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

22:32:08.064 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.

22:32:08.073 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

22:32:08.074 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:

22:32:08.081 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Could not determine java version from '11.0.18'.

22:32:08.081 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

22:32:08.082 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try:

22:32:08.082 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace.

22:32:08.083 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

22:32:08.083 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Get more help at https://help.gradle.org

I have a gradle version 7.2 at systemlevel and openjdk version 11.0.18

Any comments on the docker commands?

You might need to use an older jvm like java8 or 9...

Note that even if you manage to upgrade the dependency of Jackson, you will be very likely affected by other critical vulnerabilities, including probably this one found in late 2021 and affecting Elasticsearch versions older than 7.16.2 and Logstash older than 6.8.22.

Maintaining (old) systems is always painful and expensive if not done regularly. But consider upgrading to latest versions and resolving the dependency instead, this will be a more sustainable approach.

Nevertheless, to add something helpful to your current state, JVM version issues are often related to wrong IDE configurations that uses other versions than the project was built for / configured for. Downgrading to old versions like JDK 8 or 9 as suggested will very likely solve the issue of building.

Remember to clear caches and rebuild the project and to look for Gradle version compatibilities (some Gradle versions might not be able to build the project from the Gradle files provided in the project due to important deprecations).

2 Likes

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.