Failed to start elastic search after tls enabled

security

(Rajeshwer rao Madoori) #1

Hi,

I am trying to enable tls between elasticsearch nodes.

I have created the certs .
I have added key.pem and chain.pem under my /elasticsearch/config.

But elastic search doest start .

ERROR:

`failed to initialize a KeyManagerFactory`

Setting look like this .

`    xpack.security.enabled: true
xpack.ml.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/tls_server/tls_server/key.pem
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/tls_server/tls_server/crt.pem`

(Tim Vernum) #2

Please post the rest of the error message. This is not enough information to solve the problem.


(Rajeshwer rao Madoori) #3

@TimV Thanks for responding please find the logs attached . I am using chain.pem and key.pem

[2018-10-22T22:11:12,231][DEBUG][o.e.x.c.s.SSLService ] [es-coordinating] using ssl settings [SSLConfiguration{keyConfig=[NONE], trustConfig=JDK trusted certs], cipherSuites=[[TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.2, TLSv1.1, TLSv1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}] [2018-10-22T22:11:12,333][DEBUG][o.e.x.c.s.SSLService ] [es-coordinating] using ssl settings [SSLConfiguration{keyConfig=[keyPath=[/usr/share/elasticsearch/config/tls_server/tls_server/key.pem], certPaths=[/usr/share/elasticsearch/config/tls_server/tls_server/crt.pem]], trustConfig=Combining Trust Config{JDK trusted certs, keyPath=[/usr/share/elasticsearch/config/tls_server/tls_server/key.pem], certPaths=[/usr/share/elasticsearch/config/tls_server/tls_server/crt.pem]}], cipherSuites=[[TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.2, TLSv1.1, TLSv1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}] [2018-10-22T22:11:12,496][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-coordinating] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]


(Rajeshwer rao Madoori) #4

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.0.jar:6.3.0] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.0.jar:6.3.0] Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0] ... 6 more Caused by: java.lang.reflect.InvocationTargetException at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:488) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0] ... 6 more


(Rajeshwer rao Madoori) #5

Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a KeyManagerFactory at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:69) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:409) ~[?:?] at java.util.HashMap.computeIfAbsent(HashMap.java:1138) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:79) ~[?:?] at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:134) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:488) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0] ... 6 more Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/config/tls_server/tls_server/key.pem at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215) ~[?:?] at java.nio.file.Files.newByteChannel(Files.java:369) ~[?:?] at java.nio.file.Files.newByteChannel(Files.java:415) ~[?:?] at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:?] at java.nio.file.Files.newInputStream(Files.java:154) ~[?:?] at java.nio.file.Files.newBufferedReader(Files.java:2830) ~[?:?] at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.readPrivateKey(PEMKeyConfig.java:100) ~[?:?] at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:61) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:409) ~[?:?] at java.util.HashMap.computeIfAbsent(HashMap.java:1138) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:459) ~[?:?] at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:79) ~[?:?] at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:134) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstance(Constructor.java:488) ~[?:?] at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:643) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:557) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:311) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0] ... 6 more


(Tim Vernum) #6

Check the permissions on your certificates and keys. It doesn't look like your elasticsearch user can read them.


(Rajeshwer rao Madoori) #7

Hi,

I have these permissions

-rw-------. 1 nobody nobody 1.7K Oct 23 03:13 key.pem

-rwxr-xr-x. 1 nobody nobody 5.2K Oct 23 03:13 crt.pem

Thanks
Rajesh


(Tim Vernum) #8

That doesn't seem right to me, unless your elasticsearch process is running as nobody (which is not usually a good idea).


(Rajeshwer rao Madoori) #9

Hi @TimV,

I have changed the certs its working fine now.
But i have multinode cluster .
Master-node coordinating-node data-node .
I have enabled tls and https on master node .
For coordinating and data-nodes to join the cluster what would be the configuration .
I have generated certs for all the nodes in cluster using same ca.cert .
Should it be fine if i add the ca.cert accross all the nodes??

`  elasticsearch.yml: |
cluster.name: cluster
node.master: false
node.data: false
node.name: es-coordinating
node.ingest: false
network.host: 0.0.0.0
http.port: 9200
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.unicast.hosts: ["es-coordinating","es-master","es-data"]
node.ml: false
xpack.security.enabled: true
xpack.ml.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/key.pem
xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs/crt.pem
xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs/CA.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  /usr/share/elasticsearch/config/certs/key.pem
xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/crt.pem
xpack.security.http.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/certs/CA.crt" ]`

(Tim Vernum) #10

The guide for setting up TLS is here:

If you have specific questions about steps that are not clear, please ask them as that will help us improve the documentation for future use.

Should it be fine if i add the ca.cert accross all the nodes??

In general, our recommendation is that the value of xpack.security.transport.ssl.certificate_authorities should be the same across the whole cluster, and contain a single CA.
(There are exceptions to this advice when doing rolling cetificate updates).


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.