Elasticsearch response order consistency issues

Elastic Stack Elasticsearch
1

I have been facing issues related to es response order which is getting reshuffled even though I am not querying on the fields (I have made them non-indexed so as to ignore them in scoring) and performing dfs_query_then_fetch query. Moreover, those non-indexed fields are updated in every hit to elasticsearch.

Any suggestions on how to resolve this issue?

2

I am not sure I understand. Can you please provide an example?

1 Like
3

Let's say there are 10 documents out of which 7 are indexed(by default) and remaining 3 are non-indexed (index: false), so now I am updating values to those non-indexed fields in each and every hit to es. At the same time, querying is happening over indexed fields as we can't query over non-indexed fields. Here comes the issue, for each query, the response order are not consistent, their order is getting changed even though the score is constant for the changed order (entity / doc).

4

What is the score for the documents returned? Are you applying any sorting to the query?

5

I am not applying any sorting to the query. only filters are being used.

6

If all the documents have the same score as only filters are used and no sorting is applied I do not think ordering is guaranteed.

7

Just to add, let's say there are 50 responses, out of which 20 have same scores, within that 20 responses, the orders are getting changed.
Even if the filters are used, all the docs have same scores, that way order is inconsistent.
@Sriram8894 FYI

8

I would recommend you specify a sort order if you want/need consistency, e.g. score first and _doc second as a tie-breaker.

9

I tried by applying sort order as well, still I am facing the same inconsistent behaviour.

10

What sort order did you apply? Maybe try score and _id?

11

It'd help if you could provide code examples of what you want to do, what you are running, and what you are seeing.

12

Let's say there are 10 fields inside a document out of which 7 fields are indexed(by default) and remaining 3 are non-indexed (index: false), so now I am updating values to those non-indexed fields in each and every hit to es. At the same time, querying is happening over indexed fields as we can't query over non-indexed fields. Here comes the issue, for each query, the response order are not consistent, their order is getting changed even though the score is constant for the changed order (entity / doc).

@Christian_Dahlqvist

13

Please show a complete example and specify how you are sorting and what the score of the matches are.

1 Like
14

2 posts were split to a new topic: Filtering and ordering

15

There is some basic misunderstanding here. All documents are indexed by definition. Within a document there are fields which may or may not be searchable. If you update a single field or put a complete new doc into the index it may or may not be immediately visible to a search, this depends upon the refresh setting. Elasticsearch is a non sql database and this loose connection between updates and searches is considered a strength, not a bug! Consider that sql requires the whole mechanism of transactions to tightly specify the coordination of updates with queries.
So, as mentioned earlier a worked example is best, from you, in order to help further.

16

@Christian_Dahlqvist Find the query below:

"sort": [{
"_geo_distance": {
"latlong": [{
"lat": 26.8756,
"lon": 80.9115
}],
"unit": "mi",
"distance_type": "arc",
"order": "asc",
"validation_method": "STRICT",
"ignore_unmapped": false
}
}]

17

Please provide a full example for reproduction including sample data, mappings and queries. It would also help if you showed the result you are getting with this sample data and highlight how this differs from what you expect/desire.

1 Like
18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.