How to manage large number of in clause values

[nmathur]

Hello,

I am performing in clause on elasticsearch NEST. But it is returning 400 if I am passing large number in clause values. In my case it is 1200 values. I am fetching large number of data and using scroll for that .. here I am sharing the code

 public IEnumerable<EmployeeDetail> GetDetails(IEnumerable<int> employeeIds)
        {
            var details = new List<EmployeeDetail>();

            var scanResults = ESClient().Search<EmployeeDetail>(s => s
                                .From(0)
                                .Size(10000)
                                .Index("myindex")
                                .Type("employeedetail")
                                .Query(q => q.Terms(t => t
                                              .Field(x => x.EmployeeId)
                                              .Terms(employeeIds))) //count for employeeIds can be 100, 500 or 1200 or more 
                                .Scroll("2s")
                              );

            if (!scanResults.IsValid)
            {
                throw new System.Web.HttpException(101, scanResults.DebugInformation);
            }

            details = scanResults.Documents.ToList();
            var scrolls = 0;
            var results = ESClient().Scroll<EmployeeDetail>("4s", scanResults.ScrollId);
            details.AddRange(results.Documents.ToList());

            while (results.Documents.Any())
            {
                results = ESClient().Scroll<EmployeeDetail>("4s", results.ScrollId);
                details.AddRange(results.Documents.ToList());
                scrolls++;
            }

            return details;
}

The query is working fine with 300+, 500+, but when number is 1000+ (1024) then I am getting this error:

Invalid NEST response built from a unsuccessful low level call on POST: /myindex/employeedetail/_search?scroll=2s # Audit trail of this API call: - [1] BadResponse: Node: http://localhost:9200/ Took: 00:00:00.0924061 # ServerError: ServerError: 400Type: search_phase_execution_exception Reason: "all shards failed" # OriginalException: System.Net.WebException: The remote server returned an error: (400) Bad Request. at System.Net.HttpWebRequest.GetResponse() at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) # Request: <Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.> # Response: <Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

Please share suggestions, how to handle this.

[nmathur]

Anyone? ... I breaked the value list and applied terms like this .. this is working

.Query(q => q.Terms(t => t .Field(x => x.EmployeeId) .Terms(employeeIds.Skip(0).Take(1024))) ||
            q.Terms(t => t .Field(x => x.EmployeeId) .Terms(employeeIds.Skip(1024).Take(1024)))
)

Now I am not able to figure out the way to pass the query dynamically with 'or' ..