Upgrade from 6.3.2 to 6.4.0 but have cert errors now and ES won't start

jamesl · August 24, 2018, 8:59am

Hello.
Just started the upgarde of my Elastic Cluster from 6.3.2 to 6.4.0 but I am getting this error message in my cluster.log file and ES isn't starting. I have x.pack configured and everything was working fine until the upgrade... Looks like a cert issue?

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.4.0.jar:6.4.0]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_181]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.0.jar:6.4.0]
... 15 more
Caused by: java.lang.IllegalStateException: Error parsing Private Key from: /etc/elasticsearch/els04.key. File did not contain a supported key format
at org.elasticsearch.xpack.core.ssl.PemUtils.readPrivateKey(PemUtils.java:99) ~[?:?]
at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.readPrivateKey(PEMKeyConfig.java:97) ~[?:?]
at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:58) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:356) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:388) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.&lt;init&gt;(SSLService.java:92) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.&lt;init&gt;(XPackPlugin.java:130) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]

at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_181]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.0.jar:6.4.0]

... 15 more

[2018-08-24T18:48:05,510][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [els04] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.0.jar:6.4.0]

at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.0.jar:6.4.0]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.0.jar:6.4.0]

Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.plugins.PluginsService.&lt;init&gt;(PluginsService.java:156) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.node.Node.&lt;init&gt;(Node.java:315) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.node.Node.&lt;init&gt;(Node.java:256) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.bootstrap.Bootstrap$5.&lt;init&gt;(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.0.jar:6.4.0]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.0.jar:6.4.0]

... 6 more

Extract of my /etc/elasticsearch/elasticsearch.yml file... relating to x.pack

xpack.ssl.key:                     /etc/elasticsearch/els04.key
xpack.ssl.certificate:             /etc/elasticsearch/els04.pem
xpack.ssl.certificate_authorities: [ "/etc/elasticsearch/elk_ca.pem" ]
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.enabled: true

jamesl · August 26, 2018, 2:51am

I recut my certs but this time left them in p12 format... Pushed them all to my ES Servers, updated /etc/elasticsearch/elasticsearch.yml contents with the below and they came good (all started with no errors)...

>     > xpack.security.transport.ssl.enabled: true
>     > xpack.security.transport.ssl.verification_mode: certificate
>     > xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/els04.p12
>     > xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/els04.p12
>     > xpack.security.enabled: true

system · September 23, 2018, 2:51am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.