Elasticsearch.py with SSL and api key getting 401 unauthorized error

[TonyLuc]

Objectives: To perform search on ElasticSearch via Python.
I have followed the example in https://elasticsearch-py.readthedocs.io/en/master/ section SSL and Authentication and I could not get it to work.
I been staring at this for sometimes, I know I am missing something but I don't know what? Help?

elasticsearch.py -> Version 7.0.4
python -> Version 3.7
platform: windows

Sample Code

# SSL client authentication using client_cert and client_key
from elasticsearch import Elasticsearch
from ssl import create_default_context

context = create_default_context(cafile=r"c:\certs\cert.pem")
es = Elasticsearch(
    ['someserver', 'otherserver'],
    api_key=(‘id’, ‘api_key’),
    scheme="https",
    port=9200,
    ssl_context=context,
)

try:
    res = es.search(index="filebeat-*", body={"size":1,"query": {"match_all": {}}})
    print("Got %d Hits:" % res['hits']['total']['value'])
    for hit in res['hits']['hits']:
        print("%(@timestamp)s %(message)s" % hit["_source"])
except Exception as esError:
    print('********')
    print(type(esError))
    print(esError.args)

Errors
(401, 'security_exception', {'error': {'root_cause': [{'type': 'security_exception', 'reason': 'missing authentication credentials for REST request [/logs-/_search]', 'header': {'WWW-Authenticate': ['Bearer realm="security"', 'ApiKey', 'Basic realm="security" charset="UTF-8"']}}], 'type': 'security_exception', 'reason': 'missing authentication credentials for REST request [/filebeat-/_search]', 'header': {'WWW-Authenticate': ['Bearer realm="security"', 'ApiKey', 'Basic realm="security" charset="UTF-8"']}}, 'status': 401})

What I observed:
The cert file is correct, I have used the same file for the elastic beat agents without any problem.
i.e. specify the in output.elasticsearch: -> ssl.certificate_authorities
To confirm, on a selected server, I commented the above entry and as expected no data were coming from the it till I corrected it.

The api key entry is correct. I can use it in Postman and it is returning the right data.

The cert file and api key combo works when I ran in via c# code using both the NEST high and low-level client.

The above sample works if I switch to using http_auth=("", "").

Ask
What am I missing?

Thanks in advance!

[TonyLuc]

Hi all,

Finally found a workaround for my problem.
I am using Elasticsearch.py Version 7.0.4, unless otherwise stated I think there is a potential bug.
Based on the document you set the API Key with the syntax api_key=(‘id’, ‘api_key’), this did not work.
I have to set the api key using the headers variable instead, i.e. headers={"Authorization": "ApiKey base64 encoded"}

Hopefully this will helpful to the community.