Zero-day-exploit in log4j2 which is part of elasticsearch

I do not understand which version is affected by the vulnerability.
Why is v7.7 affected while v7.8 is not?
Are they the same when using JDK version 11?

Does anyone understand?

I think you're misinterpreting the announcement which says:

Supported versions of Elasticsearch (6.8.9+, 7.8+) used with recent versions of the JDK (JDK9+) are not susceptible to either remote code execution or information leakage.

It doesn't say that 7.7 is affected, just that it's not a supported version (i.e. it's past EOL) so it's out of scope.

2021-12-16 edit to add: "out of scope" meaning "out of the scope of this particular sentence". There are other parts of the announcement that relate to EOL versions.

2 Likes

Aah! That's what I'm talking about!
I understand now.
Thank you, Mr. DavidTurner.

When will the logstash version 6.8.21 be available for download? As stated in an earlier comment, there is still a 404 error on the site of the 6.8.21 (and also the 7.16.1) version!

1 Like

Hi there,

I upgraded to 7.16.1 on my ECK cluster with the NoLookup flag in our env variables. However, I am still getting indicators from our reports that there are still files with log4j v2.11. Is there anything else we can do?

Note: the message below indicates v7.15.1 of ES but after upgrading to v7.16.1 we get something similar. Also, this filepath is also in question /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar

The library `org.apache.logging.log4j:log4j-core` version `2.11.1` was detected in `Maven library manager` located at `/var/lib/kubelet/pods/<id>/volumes/kubernetes.io~empty-dir/elastic-internal-elasticsearch-bin-local/elasticsearch-sql-cli-7.15.1.jar` and is vulnerable to `CVE-2021-44228`, which exists in versions `< 2.15.0-rc2`.

The vulnerability was found in the [Github Security Advisory](https://nvd.nist.gov/vuln/detail/CVE-2021-44228) with vendor severity: `Critical`.

The vulnerability can be remediated by updating the library to version `2.15.0-rc2` or higher, using `mvn versions:use-latest-releases -Dincludes=org.apache.logging.log4j:log4j-core`.
2 Likes

Yes. It does need to be restarted. Additionally, you might need to change the ownership of jar to logtash:logstash or whatever it was before, in case it got changed while updating the jar by removing the class.

1 Like

I am also with the newest version of Elasticsearch receiving the following
/usr/share/Elasticsearch/bin/Elasticsearch-sql-cli-7.16.1.jar contains Log4J-2.x >= 2.10.0 VULNERABLE :frowning:
/usr/share/Elasticsearch/lib/Elasticsearch-log4j-7.16.1.jar contains Log4J-2.x <= 2.0-beta8 POTENTIALLY_SAFE :expressionless: (or did you already remove JndiLookup.class?)

Anyone aware of a fix for the sql-cli or does this not pertain to most users?

If you are using bash and the **/* does NOT work, then run shopt -s globstar before running the zip command.

shopt -s globstar
ls -lrt /usr/share/logstash/logstash-core/**/*/log4j-core-2.*
zip -d <output_of_above_command> org/apache/logging/log4j/core/lookup/JndiLookup.class
chown logstash:logstash <output_of_above_command>

Restart Logstash.

Can I download the latest jars of Log4j and palace it under below folders and start Elasticsearch?
Because when I did this getting error and Elasticsearch is not starting.
usr/share/Elasticsearch/lib/log4j-api-2.11.1.jar
/usr/share/Elasticsearch/modules/repository-url/log4j-1.2-api-2.11.1.jar
/usr/share/Elasticsearch/modules/x-pack-core/log4j-1.2-api-2.11.1.jar
/usr/share/Elasticsearch/modules/x-pack-identity-provider/log4j-slf4j-impl-2.11.1.jar
/usr/share/Elasticsearch/modules/x-pack-security/log4j-slf4j-impl-2.11.1.jar
/usr/share/Elasticsearch/modules/vector-tile/log4j-slf4j-impl-2.11.1.jar

Probably permissions issue. That's why Elasticsearch is not starting. If you have removed class from the jar, check the permissions of the jar and make sure it is elasticsearch:elasticsearch or whatever it was before.

Getting this error, I checked all the files are having correct permissions

systemctl status elasticsearch.service

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2021-12-14 15:46:01 CST; 11s ago
Docs: https://www.elastic.co
Process: 27692 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 27692 (code=exited, status=1/FAILURE)

Dec 14 15:46:00 systemd-entrypoint[27692]: Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Level
Dec 14 15:46:00 systemd-entrypoint[27692]: at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
Dec 14 15:46:00 systemd-entrypoint[27692]: at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
Dec 14 15:46:00 systemd-entrypoint[27692]: at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
Dec 14 15:46:00 systemd-entrypoint[27692]: ... 3 more
Dec 14 15:46:01 systemd-entrypoint[27692]: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/status/StatusListener
Dec 14 15:46:01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Dec 14 15:46:01 systemd[1]: Failed to start Elasticsearch.
Dec 14 15:46:01 systemd[1]: Unit elasticsearch.service entered failed state.
Dec 14 15:46:01 systemd[1]: elasticsearch.service failed.

We are using Elasticsearch 7.14.1. With this version we have log4j-api-2.11.1.jar and log4j-core-2.11.1.jar, being shipped to customers. Apart from the above mentioned workaround and possible fixes, is it possible to simply replace log4j-api-2.11.1.jar with log4j-api-2.16.0.jar and log4j-core-2.11.1.jar with log4j-core-2.16.0.jar to completely remove the vulnerable jar file? Would you recommend that also as one fix apart from the above workarounds ? This is because customers are concerned of having some vulnerable components on their systems and are more interested in removing the problem itself(here the log4j 2.11 jar), rather than having some workaround or fixes with the vulnerable component.

2 Likes

Is the Elastic team aware of the new CVE-2021-45046?

Yes. They are. Their advisory does mention it at the very top.

1 Like

Is there more information on what makes Elasticsearch 5 vulnerable? We're having difficulty duplicating the vulnerability and the version of log4j in question appears (appears) to not be vulnerable: elasticsearch/apache-log4j-extras-DEPENDENCIES at 5.6 · elastic/elasticsearch · GitHub

Seems like I was looking at a cached page. Thanks.

@sandeepkanabar any idea on this ?

Hi @Mirang_Parikh - I don't know the answer to your question. But this is something you can verify with a quick POC. Try upgrading the log4j jars and make sure permissions and owner is correctly set and then do a few CRUD operations and see if all works. You might want to look at this though.

But the ideal scenario is to follow Elastic advisory. As per the updated advisory
We have released Elasticsearch 7.16.1 and 6.8.21 which contain the JVM property by default and remove certain components of Log4j out of an abundance of caution. This is applicable to both CVE-2021-44228 and CVE-2021-45046.

@sandeepkanabar I tried that and observed that Elasticsearch 7.14.1/7.15.0 does not startup cleanly. We get following errors:

[2021-12-15T14:02:22,438][ERROR][o.e.b.Bootstrap ] [SK-FS-2K19] Exception
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2049) ~[?:?]
at java.lang.ClassLoader.getParent(ClassLoader.java:1799) ~[?:?]
at org.apache.logging.log4j.util.LoaderUtil.getClassLoaders(LoaderUtil.java:113) ~[log4j-api-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.getEventServices(WatchManager.java:160) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.<init>(WatchManager.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:46) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:768) ~[log4j-core-2.16.0.jar:2.16.0]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.<init>(EcsJsonLayout.java:47) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.createLayout(EcsJsonLayout.java:124) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout$Builder.build(EcsJsonLayout.java:150) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent.<init>(DeprecationIndexingComponent.java:70) ~[?:?]
at org.elasticsearch.xpack.deprecation.Deprecation.createComponents(Deprecation.java:83) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:615) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:619) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.14.1.jar:7.14.1]
[2021-12-15T14:02:22,454][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SK-FS-2K19] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.1.jar:7.14.1]
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2049) ~[?:?]
at java.lang.ClassLoader.getParent(ClassLoader.java:1799) ~[?:?]
at org.apache.logging.log4j.util.LoaderUtil.getClassLoaders(LoaderUtil.java:113) ~[log4j-api-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.getEventServices(WatchManager.java:160) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.<init>(WatchManager.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:46) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:768) ~[log4j-core-2.16.0.jar:2.16.0]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.<init>(EcsJsonLayout.java:47) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.createLayout(EcsJsonLayout.java:124) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout$Builder.build(EcsJsonLayout.java:150) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent.<init>(DeprecationIndexingComponent.java:70) ~[?:?]
at org.elasticsearch.xpack.deprecation.Deprecation.createComponents(Deprecation.java:83) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:615) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:619) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.14.1.jar:7.14.1]
... 6 more

Do you have any idea what could be reason behind them?

The same steps do work on 7.12.0, but that is not an option we can use. Our application already integrates 7.14.1.

Can you share the output of ls -l /usr/share/elasticsearch/lib/ or ls -l <your_es_install_dir/lib