Writing data to Elasticsearch 8 over NGINX reverse proxy fails when Python code running on Ubuntu

Elastic Stack Elasticsearch
1

I have an Elasticsearch 8.12 cluster on an internal network. I use self certified certificates and everything is running well, I push data into it all the time using Python, no errors and no warnings.

I also have an Nginx reverse proxy with a proper SSL certificate which I use to provide access to Kibana from outside our network, all have been worked well for years.

I recently needed to ingest data into Elasticsearch from outside our network, so I thought of doing it over the reverse proxy. I added a proxy pass to my Elasticsearch in the reverse proxy configuration file.

I used the same Python 3.10 code that I use internally but merely replaced the URL in the code from eg. https://10.10.10.20:9200 to https://example.com/myelasticsearch.

If I run the code outside our network on a Windows11 machine, all works well, no errors and no warnings. If I run the same code on Ubunu 22.04 with Python 3.10 I get the following error: elastic_transport.TlsError: TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)).

I have scoured the Internet for solutions, tried different suggestions with the proxy settings and and I have checked and checked the code, and the certificate. I tried writing to a different server in the cluster (after I change the proxy pass of course). The same results, the code runs with no errors on Windows, and gives the same error on Ubuntu.

I have tried the same code on WSL (Ubuntu 22.04) and I still get the same error. I downgraded urllib to 1.24, although Elasticsearch complained, I still got the same error.

I changed the Elasticsearch connection code from:

elasticsearch = Elasticsearch([server_id], basic_auth=(user_name, elastic_pass), verify_certs=True, ca_certs=certificate_fullpath)

to:

context = create_default_context(cafile=certificate_fullpath) 
context.check_hostname = False
context.hostname_checks_common_name = False
elasticsearch = Elasticsearch([server_id], ssl_context=context, basic_auth=(user_name, elastic_pass))

Same issue, it works when the code is running on Windows and fails when the code is running on Ubuntu.

I added various thing to the Nginx configuration, but to no avail.

If I change verify_certs=True to verify_certs=False in the Elasticsearch connection, then data gets pushed into Elasticsearch when the code is running on Ubuntu but I get a warning that the connection is insecure.

Has anyone come across this and have they been able to solve it?

P.S: I am pretty sure a year or more ago I did push data into Elasticsearch over NGINX reverse proxy using Ubuntu without issues.

Thanks.

2

Did you test on the Ubuntu server using curl?

Can you test a request to get https://example.com/myelasticsearch using curl with the -vvvv parameter?

Did you install the ca-certificates package in your ubuntu and the python library certifi? Your issue has nothing to do with Elasticsearch and maybe not even Nginx, it is a common issue with certificates in Python.

Normally installing the ca-certificates package and the certifi lib will solve it.

3

Thank you for the response and advice, greatly appreciated.
ca-certificates package installed and updated just in case, python certifi library is also installed and up to date. I have tried all of that but to no avail. Again this only occurs when the python code is running on Ubuntu and connecting over a reverse proxy.
If I use Ubuntu internally within our private network all is fine and no issues. If I run my code on Windows11 from outside our network, then all is fine too.

I have an Arch Linux machine, I have tried it and the exact same issue, all works fine on internal network and it fails with same error if I connect over reverse proxy.

As to curl here is what I tried:

curl -k -u elastic 'https://example.com/myelasticsearch/_cat/aliases'

When I supress certificates all works.

Then I tried:

curl --cacert /home/myfolder/myca.pem  -u elastic 'https://example.com/myelasticsearch/_cat/aliases'

I got the same error mor error less:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Then I tried curl with -vvvv as you suggested:

Connected to example.com (18x.x.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I masked the IP of example.com, the certificate folder in this message is that for example.com which is a purchased and valid certificate bought from a proper CA. However, the line in the above message "TLS alert, unknown CA (560):" is odd.

If I check the certificate for example.com on Firefox, all is fine, and Firefox can see the CA authority etc..

Any ideas please?
Thank you.

4

For those who might encounter the same problem, I found the issue. It is not related to Elasticsearch as Leandrojmp said, it is to do with ca-certificates on Ubuntu. I found that the certificate authority from whom my company purchased the SSL certificate for the reverse proxy was not in the ca-certifficate package for Ubuntu. I had to add it manually to Ubuntu, then all worked after that.