AuthorizationException with python API to remote server

[yuv_c]

I am trying to use the elasticsearch python API but I get an AuthorizationException every time I run the query.

Here is the code:

host = "https:/foo.com/es" 
client = Elasticsearch(host, api_key='...MY_API_KEY...)
 
resp = client.search(
        index = "/some_index/_search",
        body = json.loads(get_query("query_name")),
    )

The response is 403 AuthorizationException.

The query returns the requested data when I run this code:

request_data = json.loads(request_data)
url = "/".join(["https:/foo.com/es" ,"some_index/_search"])
res = requests.get(
  url,
  json=request_data,
  timeout=(60, 60),
  headers={
  "Authorization": "...MY_API_KEY..."
   },
 )
res.raise_for_status()
return res.json()

Any ideas why it doesn't work with the elasticsearch API?

[warkolm (Mark Walkom)]

What version of things are you using?

[yuv_c]

First - I'm sorry but I can't tell which version of ES the server is running (it's in a different part of my organisation and a different country).
When I make the request to find out the basic info of it I get a 403 Error.

I solved my first problem, but came across a different problem.
First - Here is the solution:

es = elasticsearch.Elasticsearch((HOST_URL))
res = es.search(
    index="some_index",
    body=request_data,
    headers={
            "Authorization": "ApiKey MY_API_KEY"
        }
) 

This returns the requested data.

However when I am trying to use the scroll method of the ES python client I'm getting strange results.

I have a code block that looks like this:

es = elasticsearch.Elasticsearch((HOST_URL))
res = es.search(
    index="some_index",
    body=request_data,
    headers={
            "Authorization": "ApiKey MY_API_KEY"
        },
    scroll='20s'
) 

#   DO STUFF WITH THE RES...

while len(res["hits"]["hits"]) > 0:
        scroll_id = res['_scroll_id'] # Could change while scrolling
        res = es.scroll(body=scroll_id, 
                        scroll='20s',
                        headers={
                            "Authorization": "ApiKey MY_API_KEY"
                                }
                       )

When the scroll() function is called I get a 403 ERROR. These are the logs:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ****.services.cloud.****.com:443

DEBUG:urllib3.connectionpool:[https://****.services.cloud.****.com:443](https://dashboard.services.cloud.****.com/) "POST /es/INDEX_NAME/_search?scroll=20s HTTP/1.1" 200 1130281

INFO:elasticsearch:POST [https://****.services.cloud.****.com:443/es/INDEX_NAME/_search?scroll=20s](https://****.services.cloud.****.com/es/INDEX_NAME/_search?scroll=20s) [status:200 request:4.632s]


WARNING:elasticsearch:DELETE [https://****.services.cloud.****.com:443/es/_search/scroll](https://****.services.cloud.****.com/es/_search/scroll) [status:403 request:0.167s]

WARNING:elasticsearch:Undecodable raw error response from server: Expecting value: line 1 column 1 (char 0)
AuthorizationException                    Traceback (most recent call last)
...............
AuthorizationException: AuthorizationException(403, 'Das ist verboten')

I checked and I get a scroll_id the way that I should.
I also use the same function (with slight modifications) with a different ES server and this works fine.

[warkolm (Mark Walkom)]

403 is an auth error, I would try the username and password with curl to check it works. If not, check the auth details with the team that is running the cluster.

[yuv_c]

I will check with them.
I tried the same code with the requests library and it gets the same error when I make the second request (first request is a success, I retrieve the data along with a scroll_id). The same code runs ok on different servers.

Since the first request is made to "some_index":
https://****.services.cloud.****.com/es/INDEX_NAME/_search?scroll=20s
and the second goes to the scroll index:
https://****.services.cloud.****.com/es/_search/scroll
I'm guessing I don't have authorization for the scroll index. Is that possible?

[warkolm (Mark Walkom)]

You may not have permission to scroll, yes.