Include & Exclude of 500k ids in a search query

Elastic Stack Elasticsearch
1

Problem statement:
We've millions of ids stored in another Id store. These ids we're generating from the application & will be as same es doc id. Here, for certain use-cases, we want to include & exclude these ids in a search query. Looked into the "ids" query not sure about the limitation there. Also, is there a plugin available that makes Elasticsearch interacts with another data store directly, fetch the ids during query execution?

ES version: 6.4

2

Welcome to our community! :smiley:

It might work, you'd be better off with a filter, but that's still a huge number.

No, there's not.

That's EOL and no longer supported. Latest is 7.17 and 8.0 is not far off being GA. You should really upgrade as a matter of urgency.

3

Thanks Mark for the reply. Any suggestion to implement this?

4

Large lists of ids will always be a challenge
You may be interested in a script and some benchmarking here that shows speed-ups under certain conditions

5

Thanks @Mark_Harwood, will go through.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.