Data inconsistency - document exists / doesn't exist on every second request

(Alex Smirnov) #1

Trying to figure this out for a few hours and can't wrap my head around this.
I have an index with type products where ID is pre-assigned for each document upon product getting added to the index.

I have a separate process that runs every minute and monitors whether there're still some products in the index that don't have sufficient data elements and does appropriate actions based on product status. I recently began to notice that this process fails every once in a while and i was finally able to pinpoint how to reproduce an issue:

We have 4 elastic nodes with 1 index, 5 shards and each shard has 1 replica.
when i try to request product - every second time it will result found: true // found: false

GET my_index/products/123456
found:true and return product data

running exact same request, through the same node will render result not found:
GET my_index/products/123456
found: false

it doesn't matter which one of the servers I try to access this doc through, every node will result in exact same behavior, where 1 time it will find the doc and every 2nd time it will not find this doc.

I'm getting exact same results when I try to use search for same document /_search?q=id:123456 and /_search?q=_id:123456 ( we have redundant property "id" in each document that represents same value as actual _id of the document )

I checked with HEAD / bigdesk / kopf plugins - they all report that all shards are in good standing. All servers are properly configured where 3 out of 4 servers are expected for MASTER failover switch

How can I identify what is wrong and how is it possible that the doc doesn't exist on every 2nd request?

Thank you for any help in advance.

(system) #2