I've had the case that's described in the "Scores are not reproducible" section of this: Getting consistent scoring.
I am using the ElasticSearch Service in AWS.
I'm going to have a large index (millions of documents), but right now, I only have about 16K.
The pipeline replaces existing documents by doing a delete_by_query and a refresh before adding replacements (some would be overwritten, some would be deleted and some would be added). Along with doing the delete_by_query and refresh, I've got "gc_deletes" : 0 set in the index.
I don't know how often it will happen, but I've already seen one occurrence where I run the exact same search multiple times and I either get one result set where the top score is one thing (and the documents are ordered according to it) or another thing (and documents are ordered likewise).
Now, I don't really have the opportunity to place a "session" identifier in the preference value - I could always place the same value (something like "abc123) in that field, but to me that seems like it might be a problem because every search that will come in will have that exact same identifier. Is that indeed a problem?
Would using the dfs_query_then _fetch handle this problem? There's a note I saw in one of the ElasticSearch documentation articles that said that it shouldn't be used in production.
Finally, is there a strategy to prevent the root cause from presenting itself in the first place, or at least greatly reduce the likelihood? I was wondering whether I need to run a _forcemerge on a regular basis to make sure all deletes are not taken into consideration when scoring happens?
Thanks for any guidance.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.