SSL Certificate error when connecting to elastic search from python

udara · May 28, 2019, 3:58am

I have set up the elastic search 7.1 in my local server under port 9650. And Signed with a self-generated CA.

We have connected kibana using the steps mentioned in the below documentation.

https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html#node-certificates

Below is the Kibana.yml configuration.

    elasticsearch.username: "elastic"
    elasticsearch.password: "testpassword"
    elasticsearch.ssl.certificate: /etc/kibana/config/client.cer
    elasticsearch.ssl.key: /etc/kibana/config/client.key
    elasticsearch.ssl.certificateAuthorities: /etc/kibana/config/client-ca.cer
    elasticsearch.ssl.verificationMode: certificate

I was able to connect to ES from kibana successfully.

Now when I use the same certificates to connect to ES from python I'm getting the below error in the python code.
ConnectionError(HTTPSConnectionPool(host='10.8.1.205', port=9650): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))) caused by: SSLError(HTTPSConnectionPool(host='10.8.1.205', port=9650): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)))

and the below error from Elastic search log.

moteAddress=/10.8.1.205:58122}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:180) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?]
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 15 more
Python code to connect to the elastic client

Below is the code I used to connect to ES from python.

self.ESClient = connections.create_connection(

hosts='10.8.1.205',
		port=9650,
		http_auth=('elastic','testpassword'),
		connection_class=RequestsHttpConnection,
		scheme='https',
		use_ssl=True,
		verify_certs=True,
		ca_cert = '/etc/kibana/config/client-ca.cer',
		client_cert = '/etc/kibana/config/client.cer',
		client_key = '/etc/kibana/config/client.key',
		timeout=self.ES_TIMEOUT)

Any help on the above issue is much appreciated.

ikakavas · May 29, 2019, 6:16am

Hi there @udara,

TLS can be used both to

"Authenticate" the server in a connection. This is what usually happens when you navigate to a website with your browser and your browser validates that the certificate that the server on the other side presents is valid and signed by a trusted CA.
Authenticate the client in the connection. This is also called "mutual TLS authentication" and it means that the client also will use a certificate and key pair in order to authenticate itself to the server.

Do you also need to do mutual TLS authentication, or are you interested in setting TLS for encrypting communications between the clients and Elasticsearch and you want the clients to authenticate themselves with username and password? I'm just trying to clarify because
setting both

elasticsearch.ssl.certificate: /etc/kibana/config/client.cer
elasticsearch.ssl.key: /etc/kibana/config/client.key

and

elasticsearch.username: "elastic"
elasticsearch.password: "testpassword"

seems unnecessary.

Now, the error that you see in your Python client, basically says that it doesn't trust the certificate that Elasticsearch is presenting for TLS. You have set

ca_cert = '/etc/kibana/config/client-ca.cer',

but this ca_cert file should be the CA cert that has signed the certificate that you have configured Elasticsearch with, and not the CA cert that has signed the client certificates.

Can you share your elasticsearch.yml too ?

udara · May 29, 2019, 9:27am

In elasticsearch.yml I have set up the following configurations.

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

The certificates were generated from the steps mentioned in the below page

But to generate a client certificate we used the steps mentioned in the below link.

under Creating a client certificate.

So I believe its the same CA that signs both certificates that are configured in ES as well as the client certs. And I think that is why kibana is working with Elasticsearch. Because both are signed with the CA elastic-stack-ca.p12.

ikakavas · May 29, 2019, 9:56am

You have enabled TLS only for the transport layer and not for the http layer in Elastisearch. You are missing : Configure TLS | Elasticsearch Guide [8.11] | Elastic

or you didn't share that part with us.

It looks like Kibana works because it connects over http, and not https. What is the value for

elasticsearch.hosts:

in kibana.yml ? We can only guess if you don't share the whole configuration in elasticsearch.yml and kibana.yml

Maybe I read the above error wrong and

Received fatal alert: unknown_ca

is Elasticsearch complaining that it can't validate the certificate that the python client is sending with client_cert

which would be possible depending on the configuration you have for TLS on the http layer of ES.

Please share all applicable parts from elasticsearch.yml and kibana.yml and do specify if you want to use mutual TLS authentication for your clients connecting to Elasticsearch and we'll get to the bottom of this.

udara · May 29, 2019, 10:21am

Hi @ikakavas

My bad for not sharing the whole ymls.

But I'm using TLS in the http as well as the transport layer as well.

the full elasticsearch.yml

node.name: node-1
cluster.initial_master_nodes: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 0.0.0.0
http.port: 9650

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12 

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12 
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12

Complete kibana.yml

elasticsearch.hosts: "https://10.8.1.205:9650"
xpack.security.enabled: true
elasticsearch.username: "elastic"
elasticsearch.password: "testpassword"


elasticsearch.ssl.certificate: /etc/kibana/config/client.cer
elasticsearch.ssl.key: /etc/kibana/config/client.key


server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/config/kibana-server/kibana-server.crt
server.ssl.key: /etc/kibana/config/kibana-server/kibana-server.key

server.host: 10.8.1.205
server.port: 8798

And both kibana > elasticsearch and browser > kibana are working on https.

ikakavas · May 30, 2019, 7:01am

Thanks for the info @udara.

Would you also please specify if you want to use TLS client authentication ? So that we don't try to solve a problem you do not have after all See my original post for a description of what TLS client (or mutual) authentication is.

If you do want to use TLS Client authentication, there are parts in your configuration missing, you also need to set up a PKI Realm.
If you don't want to use TLS client authentication, then you have extra configuration that you don't need , i.e.

elasticsearch.ssl.certificate: /etc/kibana/config/client.cer
elasticsearch.ssl.key: /etc/kibana/config/client.key

in kibana.yml

But focusing in the problem at hand (unknown_ca) , can you share the output of

openssl s_client -showcerts -host 10.8.1.205 -port 9650

and

curl -v 'https://10.8.1.205:9650' --cacert client-ca.cer

system · June 27, 2019, 7:03am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Facing issue while configuring ssl for elastic search Elasticsearch elastic-stack-security	4	608	January 8, 2020
SSL Error in Kibana logs Kibana	15	1793	October 14, 2020
Kibana not connecting after https setup Kibana	7	2426	June 4, 2021
Unable to get local issuer certificate (_ssl.c:1007))) Elasticsearch elastic-stack-security , docker	2	142	August 7, 2024
Kibana throws error while connecting to elastic nodes after enabling SSL/https on elastic nodes Kibana	9	9960	March 7, 2018

SSL Certificate error when connecting to elastic search from python

Related Topics