Failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

Elastic Stack Elasticsearch
1

Getting the following exception when I try to enable the native ssl cert for es. I have check the permission for "/etc/identity/ca/cacerts.pem". It was also used for envoy proxy and it is working fine with envoy. Any ideas?

[2020-08-04T18:16:36,208][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-data-annotations-0] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.1.jar:6.8.1]

at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.1.jar:6.8.1]

Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]

... 6 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]

at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]

... 6 more

Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory

at org.elasticsearch.xpack.core.ssl.PEMTrustConfig.createTrustManager(PEMTrustConfig.java:48) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]

at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_212]

at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]

at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_212]

at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]

at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]

at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]

... 6 more

Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/identity/ca/cacerts.pem" "read")

at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_212]

at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_212]

at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_212]

at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_212]

at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:245) ~[?:?]

at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:136) ~[?:?]

at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:148) ~[?:?]

at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:212) ~[?:?]

at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_212]

at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_212]

at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_212]

at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_212]

at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:93) ~[?:?]

at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:86) ~[?:?]

at org.elasticsearch.xpack.core.ssl.PEMTrustConfig.createTrustManager(PEMTrustConfig.java:45) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]

at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_212]

at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:448) ~[?:?]

at java.util.HashMap.forEach(HashMap.java:1289) ~[?:1.8.0_212]

at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:436) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]

at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]

at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]

... 6 more
2

Hi @junhuangli I assume it is the http client communication you want to protect with TLS. It that case you have to store the CA certificates within the Elasticsearch configuration directory, see https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html#tls-http.

3

Thanks @fgjensen

Actually, I want to have tcp protected with tls. And I will use envoy for http protection.

The certs are maintained/rotated by other teams, so copying them into the config directory might be kind of troublesome. Any suggestions?

4

For security reasons, the Elasticsearch process is limited in which directories it can read from at runtime. The certificates must be in the config directory in order to be read successfully.

5

Will change the java policy config(to grant the permission for cert files access) work here or it is still not acceptable?

6

I would suggest your organisation to set up at certificate packages repository and distribute the certificates with apt (and Chocolatey on Windows). I know this takes some time to set up, if your do not have an infrastructure for handling your organisation's certificates. Anyway, it is an needed investment, since TLS certificates signed by a CA expires within 2 years.

7

Hi @fgjensen

Thanks for the suggestion

Yes, we do have a pki team handles the certs. And the certs are in a share location for all other teams to use.

I can copy them to es config every time I redeploy es, but what if the cert rotation happens afterwards when the es is already running?

8

Technically, you could make it work, but it would not be supported.

9 
 but what if the cert rotation happens afterwards when the es is already running? Elasticsearch, Logstash, Kibana and the beats listen for certificate changes, so the components will reload the certificates after a redeploy without any interruption.
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.