Inconsistent results while querying on a index

We are using elasticsearch 1.6.0 with 6 nodes.

We are getting inconsistent results while searching when the same query is requested concurrently, the order of results are changing.

Results are entirely different for search from 80 and size 2 and another search for the same query from 81 and size 2. While the 81st result should appear in both searches.

This doesn't occur when search is made only on primary shards. Replicas become inconsistent in a while.

We faced this same issue from the versions 0.90.x as well, This issue is on many versions. We are facing the same issue again on 1.6.0. On earlier versions, we solved it by making the index replica to 0 and then increasing to desired number. But since this is very risky to do it on indices of production environment, please suggest any other alternative solution to fix this inconsistency ASAP, without much risk on production cluster.

Also, Can you please fix these inconsistency issues in future release?

Thanks.

What sort order are you using? When the sort value is the same on two documents, then the order in which documents are returned might different based on the queried shard. In order to make this issue less visible, you can make each user always use the same preference so that the same shard copy will be queried. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-preference.html

@jpountz, Thanks for the reply.

"What sort order are you using? " - I experienced this without using any kind of sorting.

"you can make each user always use the same preference so that the same shard copy will be queried." - I was thinking of using _primary_first preference for all calls, does it have any impact on performance when compared to calling without preference?

Using primary_first for all calls makes little sense if you have replicas as it will prevent elasticsearch from using replicas to scale reads. It would be better to use eg. the user id as a preference for each search request.

Yes! this makes sense, a question on how userId/sessionId as preference works. Say I'm using sessionId as preference, when the user searches, it hits some shards and gets results. on another search the same shards will be used for getting results for this user.

During this time, if a new document is indexed and stored in another shard different from the user's pre-allocated shards. This newly indexed documents will not be returned on search right? Yeah, this will not cause much issue as the sessionId differs and will be searched on different shards, each time a user logins'. But would like to know how this works, please enlighten me. Thanks.

I think you are confusing preference and routing.

Say you have an index with 2 primary shards, that have one replica shard each. This is a total of 4 shards. Let's call the primary shards 0P and 1P, and the replica shards 0R and 1R.

Routing by user id would allow you to make sure that all documents from the same user id will be on the same shard. For instance maybe all documents from user foo would be on shards 0P and 0R while all documents from user bar would end up on shards 1P and 1R. So at search time, if you know which user you are searching on, you can configure routing and only go to the shard that holds data for this user.

On the contrary, preference doesn't do anything at index time. But at search time it will make sure that you always hit the same shards if you specify the same preference. Without preference the first search request might go to shards 0P and 1R while the second search request will go to 0R and 1R. With preference configured, you are guaranteed to always hit the same shards.

@Andrien, Yes I understand routing and preference are different.
I think it would be better to understand if I give you an example of how we use elasticsearch, lets say we use it for shopping cart where user searches for all of our contents/products in our website search page. The sequence of the products displayed gets changed on concurrent requests. This is the kind of issue we are facing.

Considering this case, I'm not talking about routing since we are not storing user specific contents. We want some users to search for all of our products.

Now,

As you said, say I have an index with 2 primary shards, that have one replica shard each. This is a total of 4 shards. Let's call the primary shards 0P and 1P, and the replica shards 0R and 1R.

Now, with user's sessionId as preference, on his first search if shards 0R and 1P are searched, and
parallely if on indexing, new products gets indexed on 0P and 1P. Before it gets replicated, if the user does a second search, would the products indexed on 0P be returned?

And it is obvious that some non sync has happened while elasticsearch replicates, why and when does this non sync of replicas happen? It would be very helpful if you can avoid this non sync issue from elasticsearch itself, so that every user can see the products in same sequence.

If you want every user to see products in the same sequence then you need to change your sort order to tie break on the _uid field, so that the sort order is deterministic even in case a different replica is queried or segments are being merged.

Adding our conversation here for reference,