Trying to figure this out for a few hours and can't wrap my head around this.
I have an index with type products where ID is pre-assigned for each document upon product getting added to the index.
I have a separate process that runs every minute and monitors whether there're still some products in the index that don't have sufficient data elements and does appropriate actions based on product status. I recently began to notice that this process fails every once in a while and i was finally able to pinpoint how to reproduce an issue:
We have 4 elastic nodes with 1 index, 5 shards and each shard has 1 replica.
when i try to request product - every second time it will result found: true // found: false
ie:
GET my_index/products/123456
found:true and return product data
running exact same request, through the same node will render result not found:
GET my_index/products/123456
found: false
it doesn't matter which one of the servers I try to access this doc through, every node will result in exact same behavior, where 1 time it will find the doc and every 2nd time it will not find this doc.
I'm getting exact same results when I try to use search for same document /_search?q=id:123456 and /_search?q=_id:123456 ( we have redundant property "id" in each document that represents same value as actual _id of the document )
I checked with HEAD / bigdesk / kopf plugins - they all report that all shards are in good standing. All servers are properly configured where 3 out of 4 servers are expected for MASTER failover switch
How can I identify what is wrong and how is it possible that the doc doesn't exist on every 2nd request?
Thank you for any help in advance.