ELK 7.12.1 version ssl and tsl certificate issue

Elastic Stack Elasticsearch
1

I have 3 nodes , in that 1 is master node and 2 are child node , while enable ssl and tsl certificate i am getting an error : below is the error.

at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.(Node.java:571) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:217) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.12.1.jar:7.12.1]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.12.1.jar:7.12.1]
[2022-11-21T05:35:55,721][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: you cannot specify a keystore and key file
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.12.1.jar:7.12.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.12.1.jar:7.12.1]
Caused by: java.lang.IllegalArgumentException: you cannot specify a keystore and key file
at org.elasticsearch.xpack.core.ssl.CertParsingUtils.createKeyConfig(CertParsingUtils.java:223) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLConfiguration.createKeyConfig(SSLConfiguration.java:159) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLConfiguration.(SSLConfiguration.java:52) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$getSSLConfigurations$4(SSLService.java:504) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.getSSLConfigurations(SSLService.java:499) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:292) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$17(Node.java:567) ~[elasticsearch-7.12.1.jar:7.12.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.(Node.java:571) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:217) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.12.1.jar:7.12.1]

Please help me on this

below is my yml file --

Use a descriptive name for your cluster:

cluster.name: elk

xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.key: certs/node1.key
xpack.security.transport.ssl.certificate: certs/node1.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.keystore.secure_password: Capgemini@1234
#xpack.security.transport.ssl.keystore.secure_password:
xpack.security.http.ssl.keystore.path: http.p12

xpack.security.enabled: true

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/node1.key
xpack.security.http.ssl.certificate: certs/node1.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.monitoring.collection.enabled: true

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: node1

Add custom attributes to the node:

#node.attr.rack: r1

node.master: true
node.data: true

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

By default Elasticsearch is only accessible on localhost. Set a different

address here to expose this node on the network:

network.host: 172.16.0.62

By default Elasticsearch listens for HTTP traffic on the first free port it

finds starting at 9200. Set a specific HTTP port here:

http.port: 9200

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.seed_hosts: ["node1.elastic.test.com","node2.elastic.test.com","node3.elastic.test.com"]

Bootstrap the cluster using an initial set of master-eligible nodes:

cluster.initial_master_nodes: ["node1.elastic.test.com"]

For more information, consult the discovery and cluster formation module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true

2

Hello! The stack trace mentions this error:

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: you cannot specify a keystore and key file

I think the error is about these two settings which cannot be set at the same time.

xpack.security.http.ssl.keystore.path: http.p12
xpack.security.http.ssl.key: certs/node1.key

In general, a Keystore contains a PrivateKeyEntry which wraps two things:

  • PrivateKey
  • Certificate Chain

If you specify a key file containing a PrivateKey, then you are specifying a PrivateKey twice. Only one of the key vs keystore settings can be used.

3

I have commented bellow two lines
#xpack.security.http.ssl.keystore.path: http.p12
#xpack.security.http.ssl.key: certs/node1.key

Still i am getting the below error

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: you cannot specify a keystore and key file
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.12.1.jar:7.12.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.12.1.jar:7.12.1]
Caused by: java.lang.IllegalArgumentException: you cannot specify a keystore and key file
at org.elasticsearch.xpack.core.ssl.CertParsingUtils.createKeyConfig(CertParsingUtils.java:223) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLConfiguration.createKeyConfig(SSLConfiguration.java:159) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLConfiguration.(SSLConfiguration.java:52) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$getSSLConfigurations$4(SSLService.java:504) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.getSSLConfigurations(SSLService.java:499) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:292) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$17(Node.java:567) ~[elasticsearch-7.12.1.jar:7.12.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.(Node.java:571) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:217) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.12.1.jar:7.12.1]

4

I think this still refers to keystores, so it should be removed. Also, passwords should go in elasticsearch-keystore.

5

I tried starting 7.12.1 with your config. I had to tweak the settings, and I added some comments. I got past the error you saw using elasticsearch.yml and elasticsearch-keystore similar to this.

elasticsearch.yml example outline (change the file paths)

cluster.name: elk
bootstrap.memory_lock: true
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true

# Transport/TLS protocol settings
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate

# Transport key/cert settings, do not mix with keystore/truststore settings due to overlap
xpack.security.transport.ssl.key: ~/example/elasticsearch-7.12.1/config/certs/transport-node1.key
xpack.security.transport.ssl.certificate: ~/example/elasticsearch-7.12.1/config/certs/transport-node1.crt
xpack.security.transport.ssl.certificate_authorities: [ ~/example/elasticsearch-7.12.1/config/certs/transport-ca.crt ]

# Transport keystore/truststore settings, do not mix with key/cert settings due to overlap
#xpack.security.transport.ssl.keystore.path: ~/example/elasticsearch-7.12.1/config/certs/transport-node1-keystore.p12
#xpack.security.transport.ssl.truststore.path: ~/example/elasticsearch-7.12.1/config/certs/transport-node1-truststore.p12

# HTTPS protocol settings
xpack.security.http.ssl.enabled: true

# HTTP key/cert settings, do not mix with Keystore/Truststore settings due to overlap
xpack.security.http.ssl.key: ~/example/elasticsearch-7.12.1/config/certs/http-node1.key
xpack.security.http.ssl.certificate: ~/example/elasticsearch-7.12.1/config/certs/http-node1.crt
xpack.security.http.ssl.certificate_authorities: [ ~/example/elasticsearch-7.12.1/config/certs/http-ca.crt ]

# HTTP keystore/truststore settings, do not mix with key/cert settings due to overlap
#xpack.security.http.ssl.keystore.path: ~/example/elasticsearch-7.12.1/config/certs/http-node1-keystore.p12
#xpack.security.http.ssl.truststore.path: ~/example/elasticsearch-7.12.1/config/certs/http-node1-truststore.p12

elasticsearch-keystore example commands (change the passwords)

bin/elasticsearch-keystore create

# Only use these settings if you chose key/cert settings in elasticsearch.yml
#echo "HTTP-PrivateKey-Password"        | bin/elasticsearch-keystore add xpack.security.http.ssl.secure_key_passphrase
#echo "Transport-PrivateKey-Password"   | bin/elasticsearch-keystore add xpack.security.transport.ssl.secure_key_passphrase

# Only use these settings if you chose keystore/truststore settings in elasticsearch.yml
echo "HTTP-Keystore-Password"        | bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
echo "HTTP-Truststore-Password"      | bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password
echo "Transport-Keystore-Password"   | bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
echo "Transport-Truststore-Password" | bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
6

I have generated crt.pem and key.pem by using the below command
openssl pkcs12 -in elastic-certificates.p12 -out newfile.crt.pem -clcerts -nokeys
openssl pkcs12 -in elastic-certificates.p12 -out newfile.key.pem -nocerts -nodes

But for ssl.certificate_authorities i need to generate ca.crt for filebeat
Could you please help me with command to install ca.crt

7

It depends how you ran elasticsearch-certutil. I am going to assume you ran elasticsearch-certutil twice like so:

bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert -ca elastic-stack-ca.p12

I ran the two commands you provided, and made a copy of the first one to export the CA cert from the first P12.

openssl pkcs12 -in elastic-stack-ca.p12 -out ca.crt.pem -clcerts -nokeys
openssl pkcs12 -in elastic-certificates.p12 -out newfile.crt.pem -clcerts -nokeys
openssl pkcs12 -in elastic-certificates.p12 -out newfile.key.pem -nocerts -nodes

I ended up with these files:

elastic-certificates.p12
elastic-stack-ca.p12
ca.crt.pem
newfile.crt.pem
newfile.key.pem
8

I have created cluster and enable ssl and tsl certificate for 7.12.1 version and i am getting below error while generating password:

[root@testing-elk1 ~]# tail -n 50 /var/log/elasticsearch/my-application.log
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:99 ) ~[elasticsearch-ssl-config-7.12.1.jar:7.12.1]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1 335) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.ja va:1232) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?: ?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Fin al.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final. jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49. Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Fina l.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49. Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java: 501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4 .1.49.Final.jar:4.1.49.Final]
... 16 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:158) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[ ?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:99 ) ~[elasticsearch-ssl-config-7.12.1.jar:7.12.1]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1 335) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.ja va:1232) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?: ?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Fin al.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final. jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49. Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Fina l.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49. Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java: 501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4 .1.49.Final.jar:4.1.49.Final]
... 16 more
[2022-11-29T16:08:10,682][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this serve r's certificate, closing connection Netty4TcpChannel{localAddress=/172.16.0.122:9300, remoteAddress=/172.16.0.12 3:48720, profile=default}

Please help me on this

9

this is the elk.yml file

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/http.p12

10

I am getting below error while setting interactive password

[root@testing-elk1 bin]# ./elasticsearch-setup-passwords interactive

Failed to determine the health of the cluster running at https://172.16.0.122:9200
Unexpected response code [503] from calling GET https://172.16.0.122:9200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

Do you want to continue with the password setup process [y/N]y

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:

Unexpected response code [503] from calling PUT https://172.16.0.122:9200/_security/user/apm_system/_password?pretty
Cause: Cluster state has not been recovered yet, cannot write to the [null] index

Possible next steps:

  • Try running this tool again.
  • Try running with the --verbose parameter for additional messages.
  • Check the elasticsearch logs for additional error details.
  • Use the change password API manually.

ERROR: Failed to set password for user [apm_system].
[root@testing-elk1 bin]#

please help me on this

11

This is a general PKIX error about configuration of a truststore. Is it possible you created the three node P12s with elasticsearch-certutil without adding the --ca-dn and -pass parameters?

There are a lot of ways to set up CAs, but I can give you a specific example. One way is to create a CA P12 first, and then use it to create three Node P12s which are signed by that CA. If you do that, you should be able to use the three P12s in your nodes as a dual keystore and truststore.

If you fix the cluster member certs to trust each other, the cluster membership errors will go away, and the cluster state error will disappear because your cluster will have a quorum (i.e. minimum 2 out of 3 nodes are running and connected).

12

Could you please share me the steps please

13

There are a lot of ways to set up CAs. For manual setup, I personally like to create a CA per cluster just for Transport certs (i.e. cluster membership), and use a different (i.e. public) CA for issuing HTTPS certs.

This example creates crt/key files for each node, signed by a CA key/crt.

tar xfz elasticsearch-8.5.1-linux-x86_64.tar.gz
cd elasticsearch-8.5.1
echo "transport-ca.zip"             | bin/elasticsearch-certutil ca   --pem --days 7320 --keysize 3072 --ca-dn   CN=transport-ca --pass transport-ca
unzip transport-ca.zip
mv ca transport-ca
echo "transport-elasticsearch1.zip" | bin/elasticsearch-certutil cert --pem --days 398  --keysize 2048 --ca-cert transport-ca/ca.crt --ca-pass transport-ca --ca-key transport-ca/ca.key --pass transport-elasticsearch1 --name transport-elasticsearch1 --dns transport-elasticsearch1 --dns `hostname` --dns localhost --dns localhost.localdomain --ip 127.0.0.1 --ip ::1
echo "transport-elasticsearch2.zip" | bin/elasticsearch-certutil cert --pem --days 398  --keysize 2048 --ca-cert transport-ca/ca.crt --ca-pass transport-ca --ca-key transport-ca/ca.key --pass transport-elasticsearch2 --name transport-elasticsearch2 --dns transport-elasticsearch2 --dns `hostname` --dns localhost --dns localhost.localdomain --ip 127.0.0.1 --ip ::1
echo "transport-elasticsearch3.zip" | bin/elasticsearch-certutil cert --pem --days 398  --keysize 2048 --ca-cert transport-ca/ca.crt --ca-pass transport-ca --ca-key transport-ca/ca.key --pass transport-elasticsearch3 --name transport-elasticsearch3 --dns transport-elasticsearch3 --dns `hostname` --dns localhost --dns localhost.localdomain --ip 127.0.0.1 --ip ::1
unzip transport-elasticsearch1.zip
unzip transport-elasticsearch2.zip
unzip transport-elasticsearch3.zip
find . -name \*.crt | xargs ls -l
find . -name \*.key | xargs ls -l

The end result of this for me is:

-rw-rw-r-- 1 q q 1468 Nov 29 12:35 ./transport-ca/ca.crt
-rw-rw-r-- 1 q q 1448 Nov 29 12:35 ./transport-elasticsearch1/transport-elasticsearch1.crt
-rw-rw-r-- 1 q q 1448 Nov 29 12:35 ./transport-elasticsearch2/transport-elasticsearch2.crt
-rw-rw-r-- 1 q q 1448 Nov 29 12:35 ./transport-elasticsearch3/transport-elasticsearch3.crt
-rw-rw-r-- 1 q q 2546 Nov 29 12:35 ./transport-ca/ca.key
-rw-rw-r-- 1 q q 1766 Nov 29 12:35 ./transport-elasticsearch1/transport-elasticsearch1.key
-rw-rw-r-- 1 q q 1766 Nov 29 12:35 ./transport-elasticsearch2/transport-elasticsearch2.key
-rw-rw-r-- 1 q q 1766 Nov 29 12:35 ./transport-elasticsearch3/transport-elasticsearch3.key
14

This example outputs P12 files in DER encoding containing PrivateKeyEntry and TrustedCertificate entries.

tar xfz elasticsearch-8.5.1-linux-x86_64.tar.gz
cd elasticsearch-8.5.1
echo "transport-ca.p12"             | bin/elasticsearch-certutil ca   --days 7320 --keysize 3072 --ca-dn CN=transport-ca        --pass transport-ca
echo "transport-elasticsearch1.p12" | bin/elasticsearch-certutil cert --days 398  --keysize 2048 --ca       transport-ca.p12 --ca-pass transport-ca --pass transport-elasticsearch1 --name transport-elasticsearch1 --dns transport-elasticsearch1 --dns `hostname` --dns localhost --dns localhost.localdomain --ip 127.0.0.1 --ip ::1
echo "transport-elasticsearch2.p12" | bin/elasticsearch-certutil cert --days 398  --keysize 2048 --ca       transport-ca.p12 --ca-pass transport-ca --pass transport-elasticsearch2 --name transport-elasticsearch2 --dns transport-elasticsearch2 --dns `hostname` --dns localhost --dns localhost.localdomain --ip 127.0.0.1 --ip ::1
echo "transport-elasticsearch3.p12" | bin/elasticsearch-certutil cert --days 398  --keysize 2048 --ca       transport-ca.p12 --ca-pass transport-ca --pass transport-elasticsearch3 --name transport-elasticsearch3 --dns transport-elasticsearch3 --dns `hostname` --dns localhost --dns localhost.localdomain --ip 127.0.0.1 --ip ::1
find . -name \*.p12 | xargs ls -l

The output of this is:

-rw------- 1 q q 3440 Nov 29 12:53 ./transport-ca.p12
-rw------- 1 q q 4076 Nov 29 12:55 ./transport-elasticsearch1.p12
-rw------- 1 q q 4076 Nov 29 12:55 ./transport-elasticsearch2.p12
-rw------- 1 q q 4076 Nov 29 12:55 ./transport-elasticsearch3.p12
15

For clarification, it is fine to use your current version 7.12.1 or a newer version 8.5.1 to generate certs for your 7.12.1 deployment. Sometimes it is desirable to use a newer elasticsearch-certutil if it supports any convenient, newer parameters.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.