Query with a large set of ids (67000+ terms)

Elastic Stack Elasticsearch
1

Hello,

I have a query for document that I want to filter by excluding a large set of ids (67000 terms) that I'm current doing as following:

    	query := map[string]interface{}{
		"query": map[string]interface{}{
			"bool": map[string]interface{}{
				"must_not": map[string]interface{}{
					"terms": map[string]interface{}{
						"uploadId": uploadIDs,
					},
				},
			},
		},
	}

uploadIDs is an array of string.

While fetching the ES documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-terms-query.html#query-dsl-terms-lookup

I see that there is a default limit in the number of terms you can pass of 65,536 terms.

And right now, while trying to process this large amount of must_not I face this result:

ES Search error {"status": "400 Bad Request", "type": "search_phase_execution_exception", "reason": "all shards failed"}

which works fine if I use, let's say half the amount of uploadIDs.

My question:
do I have to modify the index.max_terms_count or is there another solution to build the request?

I'm not entirely sure I understand the doc on this concern, any ideas would be great.

Many thanks,
Jc

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.