Zero-day-exploit in log4j2 which is part of elasticsearch

Can I download the latest jars of Log4j and palace it under below folders and start Elasticsearch?
Because when I did this getting error and Elasticsearch is not starting.
usr/share/Elasticsearch/lib/log4j-api-2.11.1.jar
/usr/share/Elasticsearch/modules/repository-url/log4j-1.2-api-2.11.1.jar
/usr/share/Elasticsearch/modules/x-pack-core/log4j-1.2-api-2.11.1.jar
/usr/share/Elasticsearch/modules/x-pack-identity-provider/log4j-slf4j-impl-2.11.1.jar
/usr/share/Elasticsearch/modules/x-pack-security/log4j-slf4j-impl-2.11.1.jar
/usr/share/Elasticsearch/modules/vector-tile/log4j-slf4j-impl-2.11.1.jar

Probably permissions issue. That's why Elasticsearch is not starting. If you have removed class from the jar, check the permissions of the jar and make sure it is elasticsearch:elasticsearch or whatever it was before.

Getting this error, I checked all the files are having correct permissions

systemctl status Elasticsearch.service

● Elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/Elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2021-12-14 15:46:01 CST; 11s ago
Docs: https://www.elastic.co
Process: 27692 ExecStart=/usr/share/Elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/Elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 27692 (code=exited, status=1/FAILURE)

Dec 14 15:46:00 systemd-entrypoint[27692]: Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Level
Dec 14 15:46:00 systemd-entrypoint[27692]: at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
Dec 14 15:46:00 systemd-entrypoint[27692]: at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
Dec 14 15:46:00 systemd-entrypoint[27692]: at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
Dec 14 15:46:00 systemd-entrypoint[27692]: ... 3 more
Dec 14 15:46:01 systemd-entrypoint[27692]: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/status/StatusListener
Dec 14 15:46:01 systemd[1]: Elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Dec 14 15:46:01 systemd[1]: Failed to start Elasticsearch.
Dec 14 15:46:01 systemd[1]: Unit Elasticsearch.service entered failed state.
Dec 14 15:46:01 systemd[1]: Elasticsearch.service failed.

We are using Elasticsearch 7.14.1. With this version we have log4j-api-2.11.1.jar and log4j-core-2.11.1.jar, being shipped to customers. Apart from the above mentioned workaround and possible fixes, is it possible to simply replace log4j-api-2.11.1.jar with log4j-api-2.16.0.jar and log4j-core-2.11.1.jar with log4j-core-2.16.0.jar to completely remove the vulnerable jar file? Would you recommend that also as one fix apart from the above workarounds ? This is because customers are concerned of having some vulnerable components on their systems and are more interested in removing the problem itself(here the log4j 2.11 jar), rather than having some workaround or fixes with the vulnerable component.

2 Likes

Is the Elastic team aware of the new CVE-2021-45046?

Yes. They are. Their advisory does mention it at the very top.

1 Like

Is there more information on what makes Elasticsearch 5 vulnerable? We're having difficulty duplicating the vulnerability and the version of log4j in question appears (appears) to not be vulnerable: elasticsearch/apache-log4j-extras-DEPENDENCIES at 5.6 · elastic/elasticsearch · GitHub

Seems like I was looking at a cached page. Thanks.

@sandeepkanabar any idea on this ?

Hi @Mirang_Parikh - I don't know the answer to your question. But this is something you can verify with a quick POC. Try upgrading the log4j jars and make sure permissions and owner is correctly set and then do a few CRUD operations and see if all works. You might want to look at this though.

But the ideal scenario is to follow Elastic advisory. As per the updated advisory
We have released Elasticsearch 7.16.1 and 6.8.21 which contain the JVM property by default and remove certain components of Log4j out of an abundance of caution. This is applicable to both CVE-2021-44228 and CVE-2021-45046.

@sandeepkanabar I tried that and observed that Elasticsearch 7.14.1/7.15.0 does not startup cleanly. We get following errors:

[2021-12-15T14:02:22,438][ERROR][o.e.b.Bootstrap ] [SK-FS-2K19] Exception
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2049) ~[?:?]
at java.lang.ClassLoader.getParent(ClassLoader.java:1799) ~[?:?]
at org.apache.logging.log4j.util.LoaderUtil.getClassLoaders(LoaderUtil.java:113) ~[log4j-api-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.getEventServices(WatchManager.java:160) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.<init>(WatchManager.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:46) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:768) ~[log4j-core-2.16.0.jar:2.16.0]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.<init>(EcsJsonLayout.java:47) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.createLayout(EcsJsonLayout.java:124) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout$Builder.build(EcsJsonLayout.java:150) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent.<init>(DeprecationIndexingComponent.java:70) ~[?:?]
at org.elasticsearch.xpack.deprecation.Deprecation.createComponents(Deprecation.java:83) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:615) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:619) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.14.1.jar:7.14.1]
[2021-12-15T14:02:22,454][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SK-FS-2K19] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.1.jar:7.14.1]
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2049) ~[?:?]
at java.lang.ClassLoader.getParent(ClassLoader.java:1799) ~[?:?]
at org.apache.logging.log4j.util.LoaderUtil.getClassLoaders(LoaderUtil.java:113) ~[log4j-api-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.getEventServices(WatchManager.java:160) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.<init>(WatchManager.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:46) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:768) ~[log4j-core-2.16.0.jar:2.16.0]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.<init>(EcsJsonLayout.java:47) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.createLayout(EcsJsonLayout.java:124) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout$Builder.build(EcsJsonLayout.java:150) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent.<init>(DeprecationIndexingComponent.java:70) ~[?:?]
at org.elasticsearch.xpack.deprecation.Deprecation.createComponents(Deprecation.java:83) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:615) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:619) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.14.1.jar:7.14.1]
... 6 more

Do you have any idea what could be reason behind them?

The same steps do work on 7.12.0, but that is not an option we can use. Our application already integrates 7.14.1.

Can you share the output of ls -l /usr/share/elasticsearch/lib/ or ls -l <your_es_install_dir/lib

We are using Windows.


 Directory of E:\elasticsearch-7.14.1-windows-x86_64\elasticsearch-7.14.1\lib

12/15/2021  06:44 PM    <DIR>          .
12/15/2021  06:44 PM    <DIR>          ..
12/15/2021  01:58 PM        13,918,253 elasticsearch-7.14.1.jar
12/15/2021  01:58 PM            25,662 elasticsearch-cli-7.14.1.jar
12/15/2021  01:58 PM            69,236 elasticsearch-core-7.14.1.jar
12/15/2021  01:58 PM            52,995 elasticsearch-geo-7.14.1.jar
12/15/2021  01:58 PM            43,717 elasticsearch-launchers-7.14.1.jar
12/15/2021  01:58 PM            14,307 elasticsearch-plugin-classloader-7.14.1.jar
12/15/2021  01:58 PM            18,561 elasticsearch-secure-sm-7.14.1.jar
12/15/2021  01:58 PM           154,954 elasticsearch-x-content-7.14.1.jar
12/15/2021  01:58 PM           114,165 HdrHistogram-2.1.9.jar
12/15/2021  01:58 PM         1,159,086 hppc-0.8.1.jar
12/15/2021  01:58 PM           349,273 jackson-core-2.10.4.jar
12/15/2021  01:58 PM            58,567 jackson-dataformat-cbor-2.10.4.jar
12/15/2021  01:58 PM            90,817 jackson-dataformat-smile-2.10.4.jar
12/15/2021  01:58 PM            46,788 jackson-dataformat-yaml-2.10.4.jar
12/15/2021  01:58 PM            16,512 java-version-checker-7.14.1.jar
12/15/2021  01:58 PM           463,971 jna-5.7.0-1.jar
12/15/2021  01:58 PM           644,419 joda-time-2.10.10.jar
12/15/2021  01:58 PM            78,074 jopt-simple-5.0.2.jar
12/15/2021  01:58 PM           797,736 jts-core-1.15.0.jar
12/12/2021  11:35 PM           301,892 log4j-api-2.16.0.jar
12/12/2021  11:35 PM         1,789,565 log4j-core-2.16.0.jar
12/15/2021  01:58 PM         1,776,205 lucene-analyzers-common-8.9.0.jar
12/15/2021  01:58 PM           155,065 lucene-backward-codecs-8.9.0.jar
12/15/2021  01:58 PM         3,579,474 lucene-core-8.9.0.jar
12/15/2021  01:58 PM            98,341 lucene-grouping-8.9.0.jar
12/15/2021  01:58 PM           209,959 lucene-highlighter-8.9.0.jar
12/15/2021  01:58 PM           152,525 lucene-join-8.9.0.jar
12/15/2021  01:58 PM            52,141 lucene-memory-8.9.0.jar
12/15/2021  01:58 PM           105,985 lucene-misc-8.9.0.jar
12/15/2021  01:58 PM           381,773 lucene-queries-8.9.0.jar
12/15/2021  01:58 PM           382,661 lucene-queryparser-8.9.0.jar
12/15/2021  01:58 PM           223,460 lucene-sandbox-8.9.0.jar
12/15/2021  01:58 PM           240,654 lucene-spatial-extras-8.9.0.jar
12/15/2021  01:58 PM           309,351 lucene-spatial3d-8.9.0.jar
12/15/2021  01:58 PM           249,877 lucene-suggest-8.9.0.jar
12/15/2021  01:58 PM           682,804 lz4-java-1.8.0.jar
12/15/2021  01:58 PM           309,001 snakeyaml-1.26.jar
12/15/2021  01:58 PM           204,833 spatial4j-0.7.jar
12/15/2021  01:58 PM            51,208 t-digest-3.2.jar
12/15/2021  01:58 PM    <DIR>          tools
              40 File(s)     29,373,867 bytes
               3 Dir(s)   8,790,441,984 bytes free

@sandeepk-veritas - As I said earlier, the best option is to upgrade to ES 7.16.1 or else add the flag.

If the ES team kept the jar, instead of upgrading it, there might be a good reason.

1 Like

Quick question; is a server safe when Elasticsearch is not running and it's ports aren't exposed to the outside world?

If ES is not running, and thereby not listening on port, you are safe.
But if it’s running and even if server is not public, it still can be vulnerable. If the query you are sending to ES contains text from user input, maybe from other applications, it can be attacked.

1 Like

I had a new question.

It doesn't say that 7.7 is affected, just that it's not a supported version (i.e. it's past EOL ) so it's out of scope.

However, in the announcement:

Most other versions (5.6.11+, 6.4.0+ and 7.0.0+) can be protected via a simple JVM property change.

Does that mean 7.0.0+ can only be protected if I do that?
Or are these protected by JDK9+?

Our advice is exactly as stated in the security announcement:

The simplest remediation is to set the JVM option -Dlog4j2.formatMsgNoLookups=true and restart each node of the cluster.
For Elasticsearch 5.6.11+, 6.4+, and 7.0+, this provides full protection against the RCE and information leak attacks.

1 Like

Thank you, Mr. TimV.
I had understood the announcement.

Supported versions of Elasticsearch (6.8.9+, 7.8+) used with recent versions of the JDK (JDK9+) are not susceptible to either remote code execution or information leakage. This is due to Elasticsearch’s usage of the Java Security Manager. Most other versions (5.6.11+, 6.4.0+ and 7.0.0+) can be protected via a simple JVM property change.

On the other hand, the announcement also says:

Elasticsearch 6 and 7 are not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager.

In addition, Mr. DavidTurner said:

It doesn't say that 7.7 is affected, just that it's not a supported version (i.e. it's past EOL ) so it's out of scope.

I don't understand why only unsupported versions need to be addressed for vulnerabilities.
What I would like to know is why there is a difference in the way v7.8 and v7.7 address this vulnerability.
Are there any technical differences between these versions for vulnerabilities?

Hi,

How we will use this command for windows.After running this command on linux I am getting below response for the files.

[*] Found CVE-2021-44228 vulnerability in /usr/share/logstash/logstash-core/lib/jars/log4j-core-2.13.3.jar, log4j 2.13.3 (mitigated)

Thanks