# Elastic Agent/Fleet Server cannot connect to elasticsearch

**URL:** <https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418>\
**Category:** Beats\
**Tags:** elastic-agent\
**Created:** [December 19, 2021, 9:40pm UTC](https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418 "2021-12-19T21:40:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\
**Post date:** [December 19, 2021, 9:40pm UTC](https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418/1 "2021-12-19T21:40:58Z")

</div>

Hello,

I recently deployed Fleet Server with this command.

```auto
sudo elastic-agent enroll --url=https://fqdn:8220 \
  --fleet-server-es=https://other-fqdn:9200 \
  --fleet-server-service-token=token \
  --fleet-server-policy=0ff85c51-5cba-11ec-a216-b7b886f3c65b \
  --certificate-authorities=/path/elastic-ca.pem \
  --fleet-server-es-ca=/path/elastic-ca.pem \
  --fleet-server-cert=/path/fleet-server.crt \
  --fleet-server-cert-key=/path/fleet-server.key

```

Cert for elastic fleet was made from the same CA as bests/kibana/Elasticsearch certs

In the Elasticsearch server's log, I can see

```auto
[2021-12-17T15:32:45,254][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58292}
[2021-12-17T15:33:16,664][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58296}
[2021-12-17T15:33:30,556][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58298}
[2021-12-17T15:33:57,313][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58300}
[2021-12-17T15:34:10,252][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58302}
[2021-12-17T15:34:52,784][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58324}
[2021-12-17T15:35:09,878][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58330}
[2021-12-17T15:35:52,714][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58344}
[2021-12-17T15:36:06,563][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58350}
[2021-12-17T15:36:36,647][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip2:9200, remoteAddress=/ip1:58358}

```

Elastic has SSL configured as bellow

```auto
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: required

```

In the documentation, I can read that

```auto

--fleet-server-es-insecure

Allows fleet server to connect to Elasticsearch in the following situations:

When connecting to an HTTP server.
When connecting to an HTTPs server and the certificate chain cannot be verified. The content is encrypted, but the certificate is not verified.
When using self-signed certificates.

When this flag is used the certificate verification is disabled.

```

I feel like using this option will not resolve this problem.  
Is the only solution for me to set Elasticsearch option  
`xpack.security.http.ssl.client_authentication: required`  
to

```auto
xpack.security.http.ssl.verification_mode: none/optional

```

I hope not. Please advice.

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [December 20, 2021, 2:09pm UTC](https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418/2 "2021-12-20T14:09:43Z")

</div>

Hi,

which Elastic stack are you using now?

---

<div class="post-metadata">

**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\
**Post date:** [December 20, 2021, 2:10pm UTC](https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418/3 "2021-12-20T14:10:33Z")

</div>

Hi,

I am on

> Name : elasticsearch  
> Version : 7.16.2

Agent is on 7.16.1

---

<div class="post-metadata">

**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\
**Post date:** [December 20, 2021, 2:16pm UTC](https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418/4 "2021-12-20T14:16:15Z")

</div>

I was testing some options

I can curl to elasticserach successfully using fleet-server.crt

When i set xpack.security.http.ssl.verification\_mode: none

I still see this log

```auto
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:357) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:203) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:298) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1344) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        ... 16 more

```

I had set fleet-server.crt in  
xpack.security.http.ssl.certificate\_authorities: [/path/fleet-server.crt]

Still, it did not help

---

<div class="post-metadata">

**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\
**Post date:** [December 21, 2021, 2:46pm UTC](https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418/5 "2021-12-21T14:46:37Z")

</div>

I was missing options:

```auto
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: required

```

when I have set them I start to see

`io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain`

I made the cert for fleet with this command

```auto

/usr/share/elasticsearch/bin/elasticsearch-certutil cert -ca /path/ca/ca.p12 -pem --ip 10.10.10.10 --fqdn.local --days 3650 --out server-fleet.zip

```

---

<div class="post-metadata">

**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\
**Post date:** [December 23, 2021, 12:51am UTC](https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418/7 "2021-12-23T00:51:15Z")

</div>

So I was trying to understand what happens here and narrow down the possible issue.  
I replaced all the certificates to have the same CA with these steps.

I made new certificates with this instances.yml file

```auto
instances:
  - name: "elk-node1"
    ip:
      - "10.10.10.3"
    dns:
      - "elastic.local"
  - name: "kibana-client"
    ip:
      - "10.10.10.2"
    dns:
      - "kibana.local"
  - name: "beats"
    ip:
      - "10.10.10.2"
    dns:
      - "beats.local"
  - name: "logstash"
    ip:
      - "10.10.10.2"
    dns:
      - "logstash.local"
  - name: "fleet-server"
    ip:
      - "10.10.10.2"
    dns:
      - "fleet.local"
  - name: "kibana-server"
    ip:
      - "10.10.10.2"
    dns:
      - "kibana.local"

```

and this command  
`/usr/share/elasticsearch/bin/elasticsearch-certutil cert --silent --in instances.yml --out elastic-certs.zip --pass somepassword --keep-ca-key --days 3650`

Then I set up elasticserach trust and key stores in elasticsearch.yml

```auto
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /path/ca.p12
xpack.security.http.ssl.keystore.path: /path/elk-node1.p12
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: required

```

added password to keystore

```auto
/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password

```

After that I set up Kibana and beats respectivily

For beats to work I used this commands for certs

```auto
openssl pkcs12 -in ca.p12 -out elastic-ca.crt -clcerts -nokeys
openssl pkcs12 -in beats.p12 -nocerts -out beats.key
openssl pkcs12 -in beats.p12 -clcerts -nokeys -out beats.crt
openssl pkcs8 -in beats.key -traditional -out plain.crt
openssl rsa -aes256 -in plain.crt -out beats.key

```

Beats and Kibana can connect and I don't see any issue.

For fleet server certificates I used these commands.

```auto
openssl pkcs12 -in ca.p12 -out elastic-ca.crt -clcerts -nokeys
openssl pkcs12 -in fleet-server.p12 -nocerts -out fleet-server.key -nodes
openssl pkcs12 -in fleet-server.p12 -clcerts -nokeys -out fleet-server.crt

```

Then after copying certs to elastic-agent folder and using this command

```auto
sudo elastic-agent enroll --url=https://fleet.local:8220 \
  --fleet-server-es=https://elastic.local:9200 \
  --fleet-server-service-token=token-from-kibana \
  --fleet-server-policy=0ff85c51-5cba-11ec-a216-b7b886f3c65b \
  --certificate-authorities=/path/elastic-ca.crt \
  --fleet-server-es-ca=/path/elastic-ca.crt \
  --fleet-server-cert=/path/fleet-server.crt \
  --fleet-server-cert-key=/path/fleet-server.key

```

This is what is seen on the fleet-server side

```auto
This will replace your current settings. Do you want to continue? [Y/n]:y
2021-12-23T00:09:53.652Z INFO cmd/enroll_cmd.go:776 Fleet Server - Stopping
2021-12-23T00:10:55.667Z INFO cmd/enroll_cmd.go:776 Fleet Server - Restarting
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.16/fleet-troubleshooting.html

```

Elasticsearch logs says then

```auto
[2021-12-23T00:09:53,670][WARN][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.10.10.3:9200, remoteAddress=/10.10.10.2:40058}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:357) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:304) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1550) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1396) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        ... 16 more

```

I was playing with

> --fleet-server-es-insecure  
> --insecure

options

as well as other certs for these flags

> --fleet-server-es-ca=  
> --fleet-server-cert=  
> --fleet-server-cert-key=

But the result was always the same.

> Empty client certificate chain

I'm just starting with an SSL certificate so maybe I'm missing something obvious? I was following documentation best as I can.

Please point me in the right direction.

---

<div class="post-metadata">

**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\
**Post date:** [December 27, 2021, 11:36am UTC](https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418/8 "2021-12-27T11:36:05Z")

</div>

Please advice.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 24, 2022, 1:37pm UTC](https://discuss.elastic.co/t/elastic-agent-fleet-server-cannot-connect-to-elasticsearch/292418/9 "2022-01-24T13:37:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.