Retrieving documents by ids in a static list using ElasticSearch

I am having a hard time trying to solve a problem. The problem is something like this -

I have millions of documents in a type called user in ElasticSearch. There are two types of lists we create with these users.

  1. Dynamic - Depending on the conditions the list members changes (for eg users which were created in last 30 days)
  2. Static - The list of users is hardcoded with CSV of user id

We query both types of the list from ElasticSearch. A dynamic list is easier to query as it has fewer conditions to evaluate and ES does that pretty well. But the problem is retrieving data from the static list as the query is a terms query which has performance issues when the number of user ids are larger.

Following is my ES query

{
    "query": {
        "terms" : { "userId" : ["userId1", "userId2"]}
    }
}

If I pass multiple userId (of the order of 50k) ES CPU spikes up and starts giving 504 error.

Is there any alternative to storing static list where the data could be retrieved quickly from ES?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.