In elasticsearch v5/6 - there is a field on a search response
hits.total. Is this field intended to be completely accurate (i.e. with a healthy cluster + no changing data should we always be able to retrieve exactly that number of hits with the relevant query)?
I can see that in version 7 of elasticsearch there is a new feature/option (track total hits) which can be enabled and may change the accuracy of the total hit count for performance reasons (but will tell you if you have received an accurate count or a lower bound). My inference from this is that pre v7, the count is intended to be reliable and exact, but I can't find this explicitly stated anywhere.
We have an application which issues a terms search to elastic (querying some keywords fields). Our application has to retrieve every single hit for the specified search, so we do this by:
- Opening a scroll for the search (noting the
- Requesting X hits from the scroll until we have retrieved exactly the number in
Recently we have experienced a cluster going into a state where the
hits.total number does not line up with the number of hits we can retrieve. In these instances, restarting the nodes fixed the problem. What we'd like to know is if this is a valid thing to happen which we need to try and deal with in our application, or if it's an error (some bad caching or corrupted cluster) which we should can just detect and throw an error for. I found one open issue on github which looks it could be what we're experiencing, except it is relating to an ids query specifically.
Thanks in advance,