Include & Exclude of 500k ids in a search query

[sanders_2345 (Sanders)]

Problem statement:
We've millions of ids stored in another Id store. These ids we're generating from the application & will be as same es doc id. Here, for certain use-cases, we want to include & exclude these ids in a search query. Looked into the "ids" query not sure about the limitation there. Also, is there a plugin available that makes Elasticsearch interacts with another data store directly, fetch the ids during query execution?

ES version: 6.4

[warkolm (Mark Walkom)]

Welcome to our community!

@sanders_2345

Looked into the "ids" query not sure about the limitation there

It might work, you'd be better off with a filter, but that's still a huge number.

@sanders_2345

Also, is there a plugin available that makes Elasticsearch interacts with another data store directly, fetch the ids during query execution?

No, there's not.

@sanders_2345

ES version: 6.4

That's EOL and no longer supported. Latest is 7.17 and 8.0 is not far off being GA. You should really upgrade as a matter of urgency.

[sanders_2345 (Sanders)]

Thanks Mark for the reply. Any suggestion to implement this?

[Mark_Harwood]

Large lists of ids will always be a challenge
You may be interested in a script and some benchmarking here that shows speed-ups under certain conditions

[sanders_2345 (Sanders)]

Thanks @Mark_Harwood, will go through.