Elasticsearch6 python client - SSL works with RequestsHttpConnection, but not with Urllib3HttpConnection

[nethlaria (james)]

Hi, having a headscratching time trying to use elasticsearch python client with client certs. Using the same client certs (they're definitely fine), I can only connect using the RequestsHttpConnection class. Here's some example code:

#! /usr/bin/python3
from elasticsearch6 import Elasticsearch, RequestsHttpConnection, Urllib3HttpConnection

client = Elasticsearch(hosts=['https://storage.k8s.myvm:30002'], #k8s node port is > 30000
        verify_certs = False, ssl_assert_hostname = False, ssl_show_warn = False,
        client_cert = 'tls.crt', client_key='tls.key',
        connection_class = Urllib3HttpConnection)  #doesn't work
        #connection_class = RequestsHttpConnection) #this works 

print(client.indices.get(index='my-write-index'))

Using Urllib3HttpConnection I just see this in the ES server logs:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain

For context I'm using Rally which seems to only support Urllib3HttpConnection . Interestingly it seems to work fine on my laptop installation but not on my Ubuntu vm (I wanted the load generator to be in the same datacenter, my laptop has about 40ms ping time) so perhaps that points to some kind of dependency issue?

[Quentin_Pradet]

What error do you get from your Python code running?

What exact version of the client are you running? Can you please try a newer version of the Python client and see if that helps? This would help us narrow down the issue.

[nethlaria (james)]

Hi Quentin, thanks for response. Client side error is quite long:

$ python test_elasticsearch.py 
/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch6/connection/http_urllib3.py:191: UserWarning: Connecting to https://<myhost>.com:30002 using SSL with verify_certs=False is insecure.
  warnings.warn(
Traceback (most recent call last):
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch6/connection/http_urllib3.py", line 232, in perform_request
    response = self.pool.urlopen(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_elasticsearch.py", line 12, in <module>
    print(client.indices.get(index='myindex'))
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch6/client/utils.py", line 101, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch6/client/indices.py", line 160, in get
    return self.transport.perform_request(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch6/transport.py", line 402, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch6/connection/http_urllib3.py", line 242, in perform_request
    raise SSLError("N/A", str(e), e)
elasticsearch6.exceptions.SSLError: ConnectionError([SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1131)) caused by: SSLError([SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1131))

These are my relevant site-packages:

ll /home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/ | grep elastic
drwxrwxr-x. 1 james james 248 Jul 25 01:05 elasticsearch
drwxrwxr-x. 1 james james 236 Jul 23 14:57 elasticsearch6
drwxrwxr-x. 1 james james 128 Jul 23 14:57 elasticsearch6-6.8.2.dist-info
drwxrwxr-x. 1 james james 468 Jul 23 14:58 elasticsearch7
drwxrwxr-x. 1 james james 126 Jul 23 14:58 elasticsearch7-7.17.4.dist-info
drwxrwxr-x. 1 james james 126 Jul 25 01:05 elasticsearch-8.3.1.dist-info
drwxrwxr-x. 1 james james 350 Jul 23 14:46 elastic_transport
drwxrwxr-x. 1 james james  96 Jul 23 14:46 elastic_transport-8.1.2.dist-info

If I switch to es7 and use Urllib3HttpConnection I still get the same error. If I switch the connection class to RequestsHttpConnection I get:

james:opensearch_py$ python test_elasticsearch.py 
Traceback (most recent call last):
  File "test_elasticsearch.py", line 12, in <module>
    print(client.indices.get(index='myindex'))
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch7/client/utils.py", line 347, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch7/client/indices.py", line 227, in get
    return self.transport.perform_request(
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch7/transport.py", line 421, in perform_request
    _ProductChecker.raise_error(self._verified_elasticsearch)
  File "/home/james/.pyenv/versions/3.8.12/lib/python3.8/site-packages/elasticsearch7/transport.py", line 638, in raise_error
    raise UnsupportedProductError(message)
elasticsearch7.exceptions.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product

But maybe that is relatively successful in that SSL handshake completed? I could try the current client version 8 but I think since my target is ES 5.6.16 (I'm trying to baseline a newer service against the old one) the gap is too big.

[nethlaria (james)]

Aha, the following does work:

#! /usr/bin/python3
from elasticsearch6 import Elasticsearch, RequestsHttpConnection, Urllib3HttpConnection

client = Elasticsearch(hosts=['https://blah.myhost:30002'],
                verify_certs = True, ssl_assert_hostname = False, ssl_show_warn = False,
                client_cert = './tls.crt', client_key='./tls.key', ca_certs='./ca.crt',
                connection_class = Urllib3HttpConnection)
                #connection_class = RequestsHttpConnection)

print(client.cluster.health())

I think it's just the logic in this section: elasticsearch-py/http_urllib3.py at 6.x · elastic/elasticsearch-py · GitHub doesn't really work anything like the same as in the equivalent http_requests.py, because you can't pass a client cert and key without also asking to verify certs. That's potentially a bug IMO - is it worth logging a bug report, do you think?

Now, I think my larger issue is in Rally after all because I get a different error there, when it tries to actually verify the servername against the CA cert, which I don't want it to do.... because they're internal k8s certs...

Thanks