Failed to start elastic search after tls enabled

Elastic Stack Elasticsearch
1

Hi,

I am trying to enable tls between elasticsearch nodes.

I have created the certs .
I have added key.pem and chain.pem under my /elasticsearch/config.

But elastic search doest start .

ERROR:

`failed to initialize a KeyManagerFactory`

Setting look like this .

`    xpack.security.enabled: true
xpack.ml.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/tls_server/tls_server/key.pem
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/tls_server/tls_server/crt.pem`
2

Please post the rest of the error message. This is not enough information to solve the problem.

3

@TimV Thanks for responding please find the logs attached . I am using chain.pem and key.pem

[2018-10-22T22:11:12,231][DEBUG][o.e.x.c.s.SSLService ] [es-coordinating] using ssl settings [SSLConfiguration{keyConfig=[NONE], trustConfig=JDK trusted certs], cipherSuites=[[TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.2, TLSv1.1, TLSv1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}] [2018-10-22T22:11:12,333][DEBUG][o.e.x.c.s.SSLService ] [es-coordinating] using ssl settings [SSLConfiguration{keyConfig=[keyPath=[/usr/share/elasticsearch/config/tls_server/tls_server/key.pem], certPaths=[/usr/share/elasticsearch/config/tls_server/tls_server/crt.pem]], trustConfig=Combining Trust Config{JDK trusted certs, keyPath=[/usr/share/elasticsearch/config/tls_server/tls_server/key.pem], certPaths=[/usr/share/elasticsearch/config/tls_server/tls_server/crt.pem]}], cipherSuites=[[TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.2, TLSv1.1, TLSv1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}] [2018-10-22T22:11:12,496][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-coordinating] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

4

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.0.jar:6.3.0] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.0.jar:6.3.0] Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0] ... 6 more Caused by: java.lang.reflect.InvocationTargetException at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:488) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0] ... 6 more

5

Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a KeyManagerFactory at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:69) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:409) ~[?:?] at java.util.HashMap.computeIfAbsent(HashMap.java:1138) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:79) ~[?:?] at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:134) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:488) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0] ... 6 more Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/config/tls_server/tls_server/key.pem at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215) ~[?:?] at java.nio.file.Files.newByteChannel(Files.java:369) ~[?:?] at java.nio.file.Files.newByteChannel(Files.java:415) ~[?:?] at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:?] at java.nio.file.Files.newInputStream(Files.java:154) ~[?:?] at java.nio.file.Files.newBufferedReader(Files.java:2830) ~[?:?] at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.readPrivateKey(PEMKeyConfig.java:100) ~[?:?] at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:61) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:409) ~[?:?] at java.util.HashMap.computeIfAbsent(HashMap.java:1138) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:79) ~[?:?] at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:134) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:488) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0] ... 6 more

6

Check the permissions on your certificates and keys. It doesn't look like your elasticsearch user can read them.

7

Hi,

I have these permissions

-rw-------. 1 nobody nobody 1.7K Oct 23 03:13 key.pem

-rwxr-xr-x. 1 nobody nobody 5.2K Oct 23 03:13 crt.pem

Thanks
Rajesh

8

That doesn't seem right to me, unless your elasticsearch process is running as nobody (which is not usually a good idea).

9

Hi @TimV,

I have changed the certs its working fine now.
But i have multinode cluster .
Master-node coordinating-node data-node .
I have enabled tls and https on master node .
For coordinating and data-nodes to join the cluster what would be the configuration .
I have generated certs for all the nodes in cluster using same ca.cert .
Should it be fine if i add the ca.cert accross all the nodes??

`  elasticsearch.yml: |
cluster.name: cluster
node.master: false
node.data: false
node.name: es-coordinating
node.ingest: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["es-coordinating","es-master","es-data"]
node.ml: false
xpack.security.enabled: true
xpack.ml.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/key.pem
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs/crt.pem
xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs/CA.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  /usr/share/elasticsearch/config/certs/key.pem
xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/crt.pem
xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs/CA.crt" ]`
10

The guide for setting up TLS is here:

If you have specific questions about steps that are not clear, please ask them as that will help us improve the documentation for future use.

Should it be fine if i add the ca.cert accross all the nodes??

In general, our recommendation is that the value of xpack.security.transport.ssl.certificate_authorities should be the same across the whole cluster, and contain a single CA.
(There are exceptions to this advice when doing rolling cetificate updates).

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.