Searching by _id returns different results when the search is repeated

[BradVido]

I'm on ES 6.8, with a 3 node cluster, and the status is green.

First of all, I'm aware that I can Get by ID, but for this scenario, I am searching by ID.

My question is.... Is this a bug, or just a quirk of how distributed search works?

This request returns a single result (as expected):

Repeating the request, returns no results:

Repeating again will return a single results, and repeating again gives no results. This pattern seems to continue forever.

An interesting note, is that when I do Get by ID for this index/type/id, my first search returns the hit quickly, and the second repeated search will hang (seemingly indefinitely) with no response.

[spinscale (Alexander Reelsen)]

Hey,

just to try things out - sounds as if one shard is not reachable and you are running into a timeout and then nothing is returned. By setting the preference parameter you can hit the same shard with a query and see if only one shard is causing that behaviour, and then take a closer look at the node hosting that shard.

--Alex

[BradVido]

I think your suspicious are correct @spinscale because after rebooting one of the nodes, all queries are now returning the same data (single doc).

What's concerning is that none of the shards reports skipped/failed...

I can no longer reproduce the issue, but if it happens again, I guess I'll have to dig in more