Fleet server and agents unable to write data to Elasticsearch - Bad Certificate

I'm getting an error when Fleet agents are sending data to Elastic. On Kibana, at the fleet tab I can see Server and Agent are reporting correctly. If I change any of the policies they get redeployed. However, no data is coming through.

ELK Stack 7.16.1

The error in the Eleasticsearch log file looks like this:

[2022-12-20T17:44:33,926][WARN ][o.e.h.AbstractHttpServerTransport] [node-01] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/elasticsearch_ip:9200, remoteAddress=/agent_ip:41744}                                             
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate                                                                                                                                                                         
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                               
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                              
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                         
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                         
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                           
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                         
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                         
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                         
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                  
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                                      
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                                
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                                     
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                                                     
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                        
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                                           
        at java.lang.Thread.run(Thread.java:833) [?:?]                                                                                                                                                                                                                                      
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate                                                                                                                                                                                                       
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]                                                                                                                                                                                                                 
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]                                                                                                                                                                                                                 
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:357) ~[?:?]                                                                                                                                                                                                        
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]                                                                                                                                                                                                              
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:203) ~[?:?]                                                                                                                                                                                                     
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]                                                                                                                                                                                                               
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]                                                                                                                                                                                                             
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]                                                                                                                                                                                                         
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]                                                                                                                                                                                                             
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]                                                                                                                                                                                                             
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]                                                                                                                                                                                                                        
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:298) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                                       
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1344) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                                                      
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                                         
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286) ~[netty-handler-4.1.66.Final.jar:4.1.66.Final]                                                                                                                                                                      
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.66.Final.jar:4.1.66.Final]
        ... 16 more                     

Fleet server is deployed with the following options:

elastic-agent enroll --url=https://fleet_server_url:8220 --fleet-server-es=https://elk_server_url:9200 --fleet-server-service-token=random_token --fleet-server-policy=random_policy --certificate-authorities=/etc/elastic-agent/ssl/ca.cert --fleet-server-es-ca=/etc/elastic-agent/ssl/elasticsearch-ca.crt --fleet-server-cert=/etc/elastic-agent/ssl/fleet.crt --fleet-server-cert-key=/etc/elastic-agent/ssl/fleet.key

Certificates are issued with certuitl:

bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --name elastic-fleet --dns elastic_fleet_url --ip server_ip --pem

When I try to connect with curl to the cluster, using the same certificates listed above, connection works fine.

root@elastic-fleet:/etc/elastic-agent/ssl# curl -k --cert fleet.crt --key leet.key --cacert ca.cert -u elastic https://elk_server_url:9200
Enter host password for user 'elastic':
{
  "name" : "node-01",
  "cluster_name" : "cluster",
  "cluster_uuid" : "W-6N8z04A",
  "version" : {
    "number" : "7.16.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "5b38441b16b1ebb16a27c107a4c3865776e20c53",
    "build_date" : "2021-12-11T00:29:38.865893768Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Elasticsearch settings:

xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12

Is there any other tests I can do apart from curl ? If I'm able to connect to elasticsearch with curl using the certificates and the ca, that I used for the Fleet server, why am I getting bad certificate ?

I've managed to get that working by disabling the SSL verification mode on the Fleet Output Settings.

ssl.verification_mode: "none"

I'll have to run it like this, until I found the reason why it wasn't verifying the certificate, or re-deploy the stack.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.