So I was trying to understand what happens here and narrow down the possible issue.
I replaced all the certificates to have the same CA with these steps.
I made new certificates with this instances.yml file
instances:
- name: "elk-node1"
ip:
- "10.10.10.3"
dns:
- "elastic.local"
- name: "kibana-client"
ip:
- "10.10.10.2"
dns:
- "kibana.local"
- name: "beats"
ip:
- "10.10.10.2"
dns:
- "beats.local"
- name: "logstash"
ip:
- "10.10.10.2"
dns:
- "logstash.local"
- name: "fleet-server"
ip:
- "10.10.10.2"
dns:
- "fleet.local"
- name: "kibana-server"
ip:
- "10.10.10.2"
dns:
- "kibana.local"
and this command
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --silent --in instances.yml --out elastic-certs.zip --pass somepassword --keep-ca-key --days 3650
Then I set up elasticserach trust and key stores in elasticsearch.yml
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /path/ca.p12
xpack.security.http.ssl.keystore.path: /path/elk-node1.p12
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: required
added password to keystore
/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password
After that I set up Kibana and beats respectivily
For beats to work I used this commands for certs
openssl pkcs12 -in ca.p12 -out elastic-ca.crt -clcerts -nokeys
openssl pkcs12 -in beats.p12 -nocerts -out beats.key
openssl pkcs12 -in beats.p12 -clcerts -nokeys -out beats.crt
openssl pkcs8 -in beats.key -traditional -out plain.crt
openssl rsa -aes256 -in plain.crt -out beats.key
Beats and Kibana can connect and I don't see any issue.
For fleet server certificates I used these commands.
openssl pkcs12 -in ca.p12 -out elastic-ca.crt -clcerts -nokeys
openssl pkcs12 -in fleet-server.p12 -nocerts -out fleet-server.key -nodes
openssl pkcs12 -in fleet-server.p12 -clcerts -nokeys -out fleet-server.crt
Then after copying certs to elastic-agent folder and using this command
sudo elastic-agent enroll --url=https://fleet.local:8220 \
--fleet-server-es=https://elastic.local:9200 \
--fleet-server-service-token=token-from-kibana \
--fleet-server-policy=0ff85c51-5cba-11ec-a216-b7b886f3c65b \
--certificate-authorities=/path/elastic-ca.crt \
--fleet-server-es-ca=/path/elastic-ca.crt \
--fleet-server-cert=/path/fleet-server.crt \
--fleet-server-cert-key=/path/fleet-server.key
This is what is seen on the fleet-server side
This will replace your current settings. Do you want to continue? [Y/n]:y
2021-12-23T00:09:53.652Z INFO cmd/enroll_cmd.go:776 Fleet Server - Stopping
2021-12-23T00:10:55.667Z INFO cmd/enroll_cmd.go:776 Fleet Server - Restarting
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.16/fleet-troubleshooting.html
Elasticsearch logs says then
[2021-12-23T00:09:53,670][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.10.10.3:9200, remoteAddress=/10.10.10.2:40058}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:357) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:304) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1550) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1396) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
... 16 more
I was playing with
--fleet-server-es-insecure
--insecure
options
as well as other certs for these flags
--fleet-server-es-ca=
--fleet-server-cert=
--fleet-server-cert-key=
But the result was always the same.
Empty client certificate chain
I'm just starting with an SSL certificate so maybe I'm missing something obvious? I was following documentation best as I can.
Please point me in the right direction.