I have an index with 40 shards and I use the timeout feature with my search query. As per the documentation, if request timeout before it completes search on all shards, it will return whatever the results other shards return. But _shard state always returns me 40 shards successful and 0 failed and 0 skipped.
Can anyone suggest how should I make a request so, I will either get correct stats or will have timed_out flag false and get correct/accurate results with timeout?
@warkolm Thanks for the reply. In my case, I know the actual count and when the timed_out flag is true, the total hits count is not accurate, and also at the same time, the shards' stats tells me all shards are successful.
@warkolm If I keep refreshing it, I will get the flag as false with one of the requests and the count that I get, that is the highest match I ever get. Also, I have indexed data in SQL, I can run the query there and it also matches the number which I get when the flag is false.
@warkolm Any suggestion from the logs? Do I need to tune in to the cluster to handle the request? 90% of traffic has a Filter size (term filter) between 1 to 25. And I also have traffic with total filters of more than 500 in many cases (~2-3%).
(Optional, Boolean) If true , returns partial results if there are shard request timeouts or shard failures. If false , returns an error with no partial results. Defaults to true .
To override the default for this field, set the search.default_allow_partial_results cluster setting to false .
@Mark_Harwood I have tried using allow_partial_results query param with my requests. But I ended up getting higher latency for my requests. Which is not acceptable in my case. I want to optimize the cluster if possible with allow_partial_results.
I'm not sure if you're setting it to true or false.
If anything I'd expect allow_partial_results:false to return faster because in the event of a timeout or failure it curtails search activity that would otherwise continue
Because I want to get an accurate search result count.
allow_partial_results:false is for returning only errors rather than partial results with warnings.
allow_partial_results:true is the behaviour typically wanted by ecommerce product searches that want to "keep on trucking" and show some results in the events of timeouts and partial failures.
allow_partial_results:false is the behaviour typically wanted by analytics e.g. summing financial figures where totals are important. A timeout or partial failure gathering data has an unbounded error margin (we don't know what we might be missing) which is unacceptable to many so better not to show anything rather than the wildly inaccurate