Python API for Elastic Search - Getting 10000 in response every time

Elastic Stack Elasticsearch
1 
client = Elasticsearch([host1, host2], http_auth=(user, password), scheme="http", port=port)
response = client.search(index="complats*", body={"from": 0, "size": 10000, "query": {
            "bool": {
                "must": [
                    {
                        "query_string": {
                            "query": "tags:\"prod\" AND severity:\"INFO\" AND service:\"abc-service\" AND msg:\"* is processed\"",
                            "fields": [],
                            "type": "best_fields",
                            "default_operator": "or",
                            "max_determinized_states": 10000,
                            "enable_position_increments": "true",
                            "fuzziness": "AUTO",
                            "fuzzy_prefix_length": 0,
                            "fuzzy_max_expansions": 50,
                            "phrase_slop": 0,
                            "escape": "false",
                            "auto_generate_synonyms_phrase_query": "true",
                            "fuzzy_transpositions": "true",
                            "boost": 1.0
                        }
                    },
                    {
                        "range": {
                            "@timestamp": {
                                "from": "now-{}s".format((now.minute + 1) * 60),
                                "to": "now",
                                "include_lower": "true",
                                "include_upper": "true",
                                "boost": 1.0
                            }
                        }
                    }
                ],
                "adjust_pure_negative": "true",
                "boost": 1.0
            }
        }})
value = response['hits']['total']['value']
print(value)

The above query is successfully connecting to elasticsearch but returning an incorrect value or 10000 every time. What could be wrong here? I've read somewhere that elasticsearch module in python has a bug where it maxes out at 10000. Changing "max_determinized_states": 10000 isn't helping and it is still returning the same value. Anyone else faced this problem? If yes how did u resolve it? Thanks in advance!

2

If you have more results then the search size (you have set it to 10000) you need to scroll over the result set.

Here is an example.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.