Elastic Agent/Fleet Server cannot connect to elasticsearch

Adriann · December 19, 2021, 9:40pm

Hello,

I recently deployed Fleet Server with this command.

sudo elastic-agent enroll --url=https://fqdn:8220 \
  --fleet-server-es=https://other-fqdn:9200 \
  --fleet-server-service-token=token \
  --fleet-server-policy=0ff85c51-5cba-11ec-a216-b7b886f3c65b \
  --certificate-authorities=/path/elastic-ca.pem \
  --fleet-server-es-ca=/path/elastic-ca.pem \
  --fleet-server-cert=/path/fleet-server.crt \
  --fleet-server-cert-key=/path/fleet-server.key

Cert for elastic fleet was made from the same CA as bests/kibana/Elasticsearch certs

In the Elasticsearch server's log, I can see

[2021-12-17T15:32:45,254][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58292}
[2021-12-17T15:33:16,664][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58296}
[2021-12-17T15:33:30,556][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58298}
[2021-12-17T15:33:57,313][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58300}
[2021-12-17T15:34:10,252][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58302}
[2021-12-17T15:34:52,784][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58324}
[2021-12-17T15:35:09,878][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58330}
[2021-12-17T15:35:52,714][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58344}
[2021-12-17T15:36:06,563][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58350}
[2021-12-17T15:36:36,647][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58358}

Elastic has SSL configured as bellow

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: required

In the documentation, I can read that


--fleet-server-es-insecure

Allows fleet server to connect to Elasticsearch in the following situations:

When connecting to an HTTP server.
When connecting to an HTTPs server and the certificate chain cannot be verified. The content is encrypted, but the certificate is not verified.
When using self-signed certificates.

When this flag is used the certificate verification is disabled.

I feel like using this option will not resolve this problem.
Is the only solution for me to set Elasticsearch option
xpack.security.http.ssl.client_authentication: required
to

xpack.security.http.ssl.verification_mode: none/optional

I hope not. Please advice.

mtojek · December 20, 2021, 2:09pm

Hi,

which Elastic stack are you using now?

Adriann · December 20, 2021, 2:10pm

Hi,

I am on

Name : elasticsearch
Version : 7.16.2

Agent is on 7.16.1

Adriann · December 20, 2021, 2:16pm

I was testing some options

I can curl to elasticserach successfully using fleet-server.crt

When i set xpack.security.http.ssl.verification_mode: none

I still see this log

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:357) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:203) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:298) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1344) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        ... 16 more

I had set fleet-server.crt in
xpack.security.http.ssl.certificate_authorities: [ /path/fleet-server.crt ]

Still, it did not help

Adriann · December 21, 2021, 2:46pm

I was missing options:

xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: required

when I have set them I start to see

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain

I made the cert for fleet with this command


/usr/share/elasticsearch/bin/elasticsearch-certutil cert -ca /path/ca/ca.p12 -pem --ip 10.10.10.10 --fqdn.local --days 3650 --out server-fleet.zip

Adriann · December 23, 2021, 12:51am

So I was trying to understand what happens here and narrow down the possible issue.
I replaced all the certificates to have the same CA with these steps.

I made new certificates with this instances.yml file

instances:
  - name: "elk-node1"
    ip:
      - "10.10.10.3"
    dns:
      - "elastic.local"
  - name: "kibana-client"
    ip:
      - "10.10.10.2"
    dns:
      - "kibana.local"
  - name: "beats"
    ip:
      - "10.10.10.2"
    dns:
      - "beats.local"
  - name: "logstash"
    ip:
      - "10.10.10.2"
    dns:
      - "logstash.local"
  - name: "fleet-server"
    ip:
      - "10.10.10.2"
    dns:
      - "fleet.local"
  - name: "kibana-server"
    ip:
      - "10.10.10.2"
    dns:
      - "kibana.local"

and this command
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --silent --in instances.yml --out elastic-certs.zip --pass somepassword --keep-ca-key --days 3650

Then I set up elasticserach trust and key stores in elasticsearch.yml

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /path/ca.p12
xpack.security.http.ssl.keystore.path: /path/elk-node1.p12
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: required

added password to keystore

/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password

After that I set up Kibana and beats respectivily

For beats to work I used this commands for certs

openssl pkcs12 -in ca.p12 -out elastic-ca.crt -clcerts -nokeys
openssl pkcs12 -in beats.p12 -nocerts -out beats.key
openssl pkcs12 -in beats.p12 -clcerts -nokeys -out beats.crt
openssl pkcs8 -in beats.key -traditional -out plain.crt
openssl rsa -aes256 -in plain.crt -out beats.key

Beats and Kibana can connect and I don't see any issue.

For fleet server certificates I used these commands.

openssl pkcs12 -in ca.p12 -out elastic-ca.crt -clcerts -nokeys
openssl pkcs12 -in fleet-server.p12 -nocerts -out fleet-server.key -nodes
openssl pkcs12 -in fleet-server.p12 -clcerts -nokeys -out fleet-server.crt

Then after copying certs to elastic-agent folder and using this command

sudo elastic-agent enroll --url=https://fleet.local:8220 \
  --fleet-server-es=https://elastic.local:9200 \
  --fleet-server-service-token=token-from-kibana \
  --fleet-server-policy=0ff85c51-5cba-11ec-a216-b7b886f3c65b \
  --certificate-authorities=/path/elastic-ca.crt \
  --fleet-server-es-ca=/path/elastic-ca.crt \
  --fleet-server-cert=/path/fleet-server.crt \
  --fleet-server-cert-key=/path/fleet-server.key

This is what is seen on the fleet-server side

This will replace your current settings. Do you want to continue? [Y/n]:y
2021-12-23T00:09:53.652Z        INFO    cmd/enroll_cmd.go:776   Fleet Server - Stopping
2021-12-23T00:10:55.667Z        INFO    cmd/enroll_cmd.go:776   Fleet Server - Restarting
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.16/fleet-troubleshooting.html

Elasticsearch logs says then

[2021-12-23T00:09:53,670][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.10.10.3:9200, remoteAddress=/10.10.10.2:40058}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:357) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:304) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1550) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1396) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        ... 16 more

I was playing with

--fleet-server-es-insecure
--insecure

options

as well as other certs for these flags

--fleet-server-es-ca=
--fleet-server-cert=
--fleet-server-cert-key=

But the result was always the same.

Empty client certificate chain

I'm just starting with an SSL certificate so maybe I'm missing something obvious? I was following documentation best as I can.

Please point me in the right direction.

Adriann · December 27, 2021, 11:36am

Please advice.

system · January 24, 2022, 1:37pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ElasticAgent enrolled with Fleet server - Connection failure Elastic Defend Elastic Agent	8	44	October 22, 2024
Fleet agent doesn't work Elastic Agent fleet	14	1016	September 13, 2022
How to deploy an elastic-agent in fleet against an elasticsearch with a self-signed certificate? Beats fleet	1	1668	July 19, 2021
Fleet Server x509 Error (GODEBUG=x509ignoreCN=0) Beats fleet , elastic-agent	5	2132	September 27, 2021
Unable to install fleet server Beats fleet	6	3951	September 23, 2021

Elastic Agent/Fleet Server cannot connect to elasticsearch

Related topics