Probably permissions issue. That's why Elasticsearch is not starting. If you have removed class from the jar, check the permissions of the jar and make sure it is elasticsearch:elasticsearch or whatever it was before.
Getting this error, I checked all the files are having correct permissions
systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2021-12-14 15:46:01 CST; 11s ago
Docs: https://www.elastic.co
Process: 27692 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 27692 (code=exited, status=1/FAILURE)
Dec 14 15:46:00 systemd-entrypoint[27692]: Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Level
Dec 14 15:46:00 systemd-entrypoint[27692]: at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
Dec 14 15:46:00 systemd-entrypoint[27692]: at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
Dec 14 15:46:00 systemd-entrypoint[27692]: at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
Dec 14 15:46:00 systemd-entrypoint[27692]: ... 3 more
Dec 14 15:46:01 systemd-entrypoint[27692]: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/status/StatusListener
Dec 14 15:46:01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Dec 14 15:46:01 systemd[1]: Failed to start Elasticsearch.
Dec 14 15:46:01 systemd[1]: Unit elasticsearch.service entered failed state.
Dec 14 15:46:01 systemd[1]: elasticsearch.service failed.
We are using Elasticsearch 7.14.1. With this version we have log4j-api-2.11.1.jar and log4j-core-2.11.1.jar, being shipped to customers. Apart from the above mentioned workaround and possible fixes, is it possible to simply replace log4j-api-2.11.1.jar with log4j-api-2.16.0.jar and log4j-core-2.11.1.jar with log4j-core-2.16.0.jar to completely remove the vulnerable jar file? Would you recommend that also as one fix apart from the above workarounds ? This is because customers are concerned of having some vulnerable components on their systems and are more interested in removing the problem itself(here the log4j 2.11 jar), rather than having some workaround or fixes with the vulnerable component.
2 Likes
Is the Elastic team aware of the new CVE-2021-45046?
Yes. They are. Their advisory does mention it at the very top.
1 Like
Is there more information on what makes Elasticsearch 5 vulnerable? We're having difficulty duplicating the vulnerability and the version of log4j in question appears (appears) to not be vulnerable: elasticsearch/apache-log4j-extras-DEPENDENCIES at 5.6 · elastic/elasticsearch · GitHub
Seems like I was looking at a cached page. Thanks.
@sandeepkanabar any idea on this ?
Hi @Mirang_Parikh - I don't know the answer to your question. But this is something you can verify with a quick POC. Try upgrading the log4j jars and make sure permissions and owner is correctly set and then do a few CRUD operations and see if all works. You might want to look at this though.
But the ideal scenario is to follow Elastic advisory. As per the updated advisory
We have released Elasticsearch 7.16.1 and 6.8.21 which contain the JVM property by default and remove certain components of Log4j out of an abundance of caution. This is applicable to both CVE-2021-44228 and CVE-2021-45046.
@sandeepkanabar I tried that and observed that Elasticsearch 7.14.1/7.15.0 does not startup cleanly. We get following errors:
[2021-12-15T14:02:22,438][ERROR][o.e.b.Bootstrap ] [SK-FS-2K19] Exception
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2049) ~[?:?]
at java.lang.ClassLoader.getParent(ClassLoader.java:1799) ~[?:?]
at org.apache.logging.log4j.util.LoaderUtil.getClassLoaders(LoaderUtil.java:113) ~[log4j-api-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.getEventServices(WatchManager.java:160) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.<init>(WatchManager.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:46) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:768) ~[log4j-core-2.16.0.jar:2.16.0]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.<init>(EcsJsonLayout.java:47) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.createLayout(EcsJsonLayout.java:124) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout$Builder.build(EcsJsonLayout.java:150) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent.<init>(DeprecationIndexingComponent.java:70) ~[?:?]
at org.elasticsearch.xpack.deprecation.Deprecation.createComponents(Deprecation.java:83) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:615) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:619) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.14.1.jar:7.14.1]
[2021-12-15T14:02:22,454][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SK-FS-2K19] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.1.jar:7.14.1]
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2049) ~[?:?]
at java.lang.ClassLoader.getParent(ClassLoader.java:1799) ~[?:?]
at org.apache.logging.log4j.util.LoaderUtil.getClassLoaders(LoaderUtil.java:113) ~[log4j-api-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.getEventServices(WatchManager.java:160) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.<init>(WatchManager.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:46) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:768) ~[log4j-core-2.16.0.jar:2.16.0]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.<init>(EcsJsonLayout.java:47) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.createLayout(EcsJsonLayout.java:124) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout$Builder.build(EcsJsonLayout.java:150) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent.<init>(DeprecationIndexingComponent.java:70) ~[?:?]
at org.elasticsearch.xpack.deprecation.Deprecation.createComponents(Deprecation.java:83) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:615) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.<init>(Node.java:619) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.node.Node.<init>(Node.java:281) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.14.1.jar:7.14.1]
... 6 more
Do you have any idea what could be reason behind them?
The same steps do work on 7.12.0, but that is not an option we can use. Our application already integrates 7.14.1.
Can you share the output of ls -l /usr/share/elasticsearch/lib/ or ls -l <your_es_install_dir/lib
We are using Windows.
Directory of E:\elasticsearch-7.14.1-windows-x86_64\elasticsearch-7.14.1\lib
12/15/2021 06:44 PM <DIR> .
12/15/2021 06:44 PM <DIR> ..
12/15/2021 01:58 PM 13,918,253 elasticsearch-7.14.1.jar
12/15/2021 01:58 PM 25,662 elasticsearch-cli-7.14.1.jar
12/15/2021 01:58 PM 69,236 elasticsearch-core-7.14.1.jar
12/15/2021 01:58 PM 52,995 elasticsearch-geo-7.14.1.jar
12/15/2021 01:58 PM 43,717 elasticsearch-launchers-7.14.1.jar
12/15/2021 01:58 PM 14,307 elasticsearch-plugin-classloader-7.14.1.jar
12/15/2021 01:58 PM 18,561 elasticsearch-secure-sm-7.14.1.jar
12/15/2021 01:58 PM 154,954 elasticsearch-x-content-7.14.1.jar
12/15/2021 01:58 PM 114,165 HdrHistogram-2.1.9.jar
12/15/2021 01:58 PM 1,159,086 hppc-0.8.1.jar
12/15/2021 01:58 PM 349,273 jackson-core-2.10.4.jar
12/15/2021 01:58 PM 58,567 jackson-dataformat-cbor-2.10.4.jar
12/15/2021 01:58 PM 90,817 jackson-dataformat-smile-2.10.4.jar
12/15/2021 01:58 PM 46,788 jackson-dataformat-yaml-2.10.4.jar
12/15/2021 01:58 PM 16,512 java-version-checker-7.14.1.jar
12/15/2021 01:58 PM 463,971 jna-5.7.0-1.jar
12/15/2021 01:58 PM 644,419 joda-time-2.10.10.jar
12/15/2021 01:58 PM 78,074 jopt-simple-5.0.2.jar
12/15/2021 01:58 PM 797,736 jts-core-1.15.0.jar
12/12/2021 11:35 PM 301,892 log4j-api-2.16.0.jar
12/12/2021 11:35 PM 1,789,565 log4j-core-2.16.0.jar
12/15/2021 01:58 PM 1,776,205 lucene-analyzers-common-8.9.0.jar
12/15/2021 01:58 PM 155,065 lucene-backward-codecs-8.9.0.jar
12/15/2021 01:58 PM 3,579,474 lucene-core-8.9.0.jar
12/15/2021 01:58 PM 98,341 lucene-grouping-8.9.0.jar
12/15/2021 01:58 PM 209,959 lucene-highlighter-8.9.0.jar
12/15/2021 01:58 PM 152,525 lucene-join-8.9.0.jar
12/15/2021 01:58 PM 52,141 lucene-memory-8.9.0.jar
12/15/2021 01:58 PM 105,985 lucene-misc-8.9.0.jar
12/15/2021 01:58 PM 381,773 lucene-queries-8.9.0.jar
12/15/2021 01:58 PM 382,661 lucene-queryparser-8.9.0.jar
12/15/2021 01:58 PM 223,460 lucene-sandbox-8.9.0.jar
12/15/2021 01:58 PM 240,654 lucene-spatial-extras-8.9.0.jar
12/15/2021 01:58 PM 309,351 lucene-spatial3d-8.9.0.jar
12/15/2021 01:58 PM 249,877 lucene-suggest-8.9.0.jar
12/15/2021 01:58 PM 682,804 lz4-java-1.8.0.jar
12/15/2021 01:58 PM 309,001 snakeyaml-1.26.jar
12/15/2021 01:58 PM 204,833 spatial4j-0.7.jar
12/15/2021 01:58 PM 51,208 t-digest-3.2.jar
12/15/2021 01:58 PM <DIR> tools
40 File(s) 29,373,867 bytes
3 Dir(s) 8,790,441,984 bytes free
@sandeepk-veritas - As I said earlier, the best option is to upgrade to ES 7.16.1 or else add the flag.
If the ES team kept the jar, instead of upgrading it, there might be a good reason.
1 Like
Quick question; is a server safe when Elasticsearch is not running and it's ports aren't exposed to the outside world?
If ES is not running, and thereby not listening on port, you are safe.
But if it’s running and even if server is not public, it still can be vulnerable. If the query you are sending to ES contains text from user input, maybe from other applications, it can be attacked.
1 Like
I had a new question.
It doesn't say that 7.7 is affected, just that it's not a supported version (i.e. it's past EOL ) so it's out of scope.
However, in the announcement:
Most other versions (5.6.11+, 6.4.0+ and 7.0.0+) can be protected via a simple JVM property change.
Does that mean 7.0.0+ can only be protected if I do that?
Or are these protected by JDK9+?
Our advice is exactly as stated in the security announcement:
The simplest remediation is to set the JVM option
-Dlog4j2.formatMsgNoLookups=trueand restart each node of the cluster.
For Elasticsearch 5.6.11+, 6.4+, and 7.0+, this provides full protection against the RCE and information leak attacks.
1 Like
Thank you, Mr. TimV.
I had understood the announcement.
Supported versions of Elasticsearch (6.8.9+, 7.8+) used with recent versions of the JDK (JDK9+) are not susceptible to either remote code execution or information leakage. This is due to Elasticsearch’s usage of the Java Security Manager. Most other versions (5.6.11+, 6.4.0+ and 7.0.0+) can be protected via a simple JVM property change.
On the other hand, the announcement also says:
Elasticsearch 6 and 7 are not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager.
In addition, Mr. DavidTurner said:
It doesn't say that 7.7 is affected, just that it's not a supported version (i.e. it's past EOL ) so it's out of scope.
I don't understand why only unsupported versions need to be addressed for vulnerabilities.
What I would like to know is why there is a difference in the way v7.8 and v7.7 address this vulnerability.
Are there any technical differences between these versions for vulnerabilities?
Hi,
How we will use this command for windows.After running this command on linux I am getting below response for the files.
[*] Found CVE-2021-44228 vulnerability in /usr/share/logstash/logstash-core/lib/jars/log4j-core-2.13.3.jar, log4j 2.13.3 (mitigated)
Thanks
It sounds like you are interpreting the announcement in ways that are not intended.
The announcement contains (at least) 2 separate things:
- details about the extent to which different versions of Elasticsearch may be affected by different parts of the log4j vulnerability
- recommended steps to take (which vary by version).
You can use part 1 to assess your risk and determine the urgency with which you need to follow the recommendations in part 2.
That is, you should follow the remediation recommendation, and either:
- Upgrade to the recently released 7.16.1 or 6.8.21 versions
- If upgrading is not possible for you, follow the other remediation steps we have outlined.
How urgently you need to do that will depend on the version you are running and how concerned you are about the risks associated with that version.
1 Like