Elastic search docker image - Jar hell error

priya_dhana · April 11, 2023, 5:18pm

I am using docker.elastic.co/elasticsearch/elasticsearch:5.6.16 as base image and trying to upgrade the jackson packages to resolve Critical CVE.

Dockerfile:

# https://github.com/elastic/elasticsearch-docker

FROM docker.elastic.co/elasticsearch/elasticsearch:5.6.16
# Remove existing jackson package
RUN rm -f /usr/share/elasticsearch/lib/jackson*


# # Installing patched Jackson packages
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.14.0/jackson-databind-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.14.0/jackson-core-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.14.0/jackson-annotations-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.14.0/jackson-dataformat-cbor-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.14.0/jackson-dataformat-yaml-2.14.0.jar -P /usr/share/elasticsearch/lib/
RUN wget https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.14.0/jackson-dataformat-smile-2.14.0.jar -P /usr/share/elasticsearch/lib/

When this image is deployed as a pod and we tried to login, the pod went to crashloop with JAR Hell error

Error:

OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[2023-04-11T08:29:49,361][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: META-INF.versions.9.module-info
jar1: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.14.0.jar
jar2: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.14.0.jar
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.16.jar:5.6.16]
Caused by: java.lang.IllegalStateException: jar hell!
class: META-INF.versions.9.module-info
jar1: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.14.0.jar
jar2: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.14.0.jar
	at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:282) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:90) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.16.jar:5.6.16]
	... 6 more

Does elasticsearch of version 5.6.16 has explicit dependency on Jackson 2.8.16 package version or we can upgrade Jackson package to 2.14.0 without compatibility issues??

Please help!!!

dadoonet · April 12, 2023, 1:26am

Welcome!

You should really think of switching to a recent version instead.

priya_dhana · April 12, 2023, 1:48am

Hi David, unfortunately we have a dependent on this particular version. We are planning to upgrade eventually. But this is an urgent requirement to patch Critical CVE. Is there any dependency with Jackson version 2.8.16 or we can upgrade the Jackson package version ?? Please help.

Thanks

dadoonet · April 12, 2023, 5:14am

It's really a matter of urgency that you upgrade your nodes.
Fixing that CVE won't fix all the security issues that you might have with this so old version.

But, back to your question. If I recall correctly how I was building Elasticsearch in the past, I'd probably:

checkout the code from GitHub
upgrade the version in the pom.xml file
compute again the signatures for all the jars (not sure if it's needed and how but the next step should tell you)
build the project

I don't think you can just replace a jar as is.

priya_dhana · April 12, 2023, 6:14am

Thank you for the response.
Since we are using the base docker image docker.elastic.co/elasticsearch/elasticsearch:5.6.16 which is pre-built. Is there any way to upgrade the version on top of the pre-built code source?

dadoonet · April 12, 2023, 12:20pm

I believe (hope) that the build does produce the image.

priya_dhana · April 13, 2023, 4:32am

Hi David, I tried to follow the steps you suggested.

Checked out github code for elasticsearch 5.6 version
There was no pom.xml file in the source code. I could only find jackson version under in buildSrc/version.properties file and updated it.
Ran -> ./gradlew localDistro to build from source code. The build failed.

Is this the correct way to upgrade jackson version? Can you please help with some docker instructions on how to upgrade this with custom dockerfile instead of updating the source code?

system · April 13, 2023, 4:32am

elasticsearch 5.6 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

dadoonet · April 13, 2023, 5:07am

Ha! I did not remember when we exactly switched to Gradle.

What is the error message in the build?

priya_dhana · April 13, 2023, 5:34am

22:32:07.944 [INFO] [org.gradle.internal.nativeintegration.services.NativeServices] Initialized native services in: /home/user/.gradle/native

22:32:08.059 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

22:32:08.064 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.

22:32:08.073 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

22:32:08.074 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:

22:32:08.081 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Could not determine java version from '11.0.18'.

22:32:08.081 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

22:32:08.082 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try:

22:32:08.082 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace.

22:32:08.083 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]

22:32:08.083 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Get more help at https://help.gradle.org

I have a gradle version 7.2 at systemlevel and openjdk version 11.0.18

Any comments on the docker commands?

dadoonet · April 13, 2023, 6:23am

You might need to use an older jvm like java8 or 9...

malliaridis · April 17, 2023, 12:55am

Note that even if you manage to upgrade the dependency of Jackson, you will be very likely affected by other critical vulnerabilities, including probably this one found in late 2021 and affecting Elasticsearch versions older than 7.16.2 and Logstash older than 6.8.22.

Maintaining (old) systems is always painful and expensive if not done regularly. But consider upgrading to latest versions and resolving the dependency instead, this will be a more sustainable approach.

Nevertheless, to add something helpful to your current state, JVM version issues are often related to wrong IDE configurations that uses other versions than the project was built for / configured for. Downgrading to old versions like JDK 8 or 9 as suggested will very likely solve the issue of building.

Remember to clear caches and rebuild the project and to look for Gradle version compatibilities (some Gradle versions might not be able to build the project from the Gradle files provided in the project due to important deprecations).

system · May 15, 2023, 12:56am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Vulnerabilities in latest docker image of Elasticsearch 8.3.1 Elasticsearch docker	2	760	August 16, 2022
Vulnerabilities in docker image elasticsearch/elasticsearch:7.17.8 Elasticsearch docker	6	803	February 15, 2023
Docker Elasticsearch 8.10.3 Java Crash Elasticsearch docker , runtime-fields	2	300	November 21, 2023
Way to change the version of JARs used in Elasticsearch Elasticsearch docker	2	785	July 13, 2021
Plugin Jar Hell Elasticsearch	3	2125	July 5, 2017

Elastic search docker image - Jar hell error

Related topics