# Very Large Terms Query

**URL:** <https://discuss.elastic.co/t/very-large-terms-query/39838>\
**Category:** Elasticsearch\
**Created:** [January 22, 2016, 12:46am UTC](https://discuss.elastic.co/t/very-large-terms-query/39838 "2016-01-22T00:46:44Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![johari](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@johari](https://discuss.elastic.co/u/johari)\
**Post date:** [January 22, 2016, 12:46am UTC](https://discuss.elastic.co/t/very-large-terms-query/39838/1 "2016-01-22T00:46:44Z")

</div>

I'm pretty new but it seems one of the most obvious applications of Elasticsearch is to filter a vast number of large documents with a terms filter containing the few words you're interested in. However, i have the inverse use case and I'm wondering if ES is as well suited to handle it. In my scenario, I have 100's of millions of small documents and I need to query to only return documents that contain at least one of 1000 words. For example, a typical indexed document might look like:  
{  
company\_id: 435,  
name: “Company X”,  
…(other fields)…,  
visibility\_terms: “word\_1, word\_2, word\_3, word\_4”  
}

and an example query may look like:  
{  
size:30,  
from:0,  
query: {  
filtered:{  
query:null,  
filter:{  
and:[{  
term:{  
company\_id:435  
},  
terms:{  
visibility\_terms:[  
..(up to 1000 unique words)..  
]  
}  
}]  
}  
}  
},  
sort: [{  
name:{  
order:"asc"  
}  
}]  
}

Something about having such a large 'or statement' feels wrong and destined to cause performance problems. Has anyone had experience issuing large terms queries like this that can shed some light on whether this is a bad idea or not?

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [January 22, 2016, 11:17am UTC](https://discuss.elastic.co/t/very-large-terms-query/39838/2 "2016-01-22T11:17:43Z")

</div>

1000 unique terms are not a problem, especially not in a filter.

There is a Lucene limit of 1024 terms in a query clause. But you could also submit a series of queries and join the result hits.

`filtered:{query:null, filter:...}` works? Interesting.

---

<div class="post-metadata">

**Author:** ![johari](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@johari](https://discuss.elastic.co/u/johari)\
**Post date:** [January 22, 2016, 8:35pm UTC](https://discuss.elastic.co/t/very-large-terms-query/39838/3 "2016-01-22T20:35:31Z")

</div>

Thanks for the reply! Yea I ran this query on a fully populated backup instance yesterday and it worked and was really performant. It's good to get some further verification that this query pattern is legit before I fully invest in this strategy. it's crazy that it can perform a 1000 line or statement so fast! ES is truly amazing 🙂

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [January 22, 2016, 11:36pm UTC](https://discuss.elastic.co/t/very-large-terms-query/39838/4 "2016-01-22T23:36:17Z")

</div>

You can achieve high performance with larger number of multi term boolean filters/queries when certain conditions can be met:

- no scoring - skipping scoring and performing constant score query with multiple filter terms saves expensive term weighting computations.
- no sorting - delivering results in the index order in they are found is faster than reordering documents
- ORing terms is faster than ANDing all terms, since not all given terms must be visited before results can be delivered

---

<div class="post-metadata">

**Author:** ![johari](https://avatars.discourse-cdn.com/v4/letter/j/e19b73/32.png) [@johari](https://discuss.elastic.co/u/johari)\
**Post date:** [January 25, 2016, 9:03pm UTC](https://discuss.elastic.co/t/very-large-terms-query/39838/5 "2016-01-25T21:03:39Z")

</div>

Thanks for the extra tips! Knowing this will definitely have an effect on the way I implement the feature. Unfortunately, some of our use cases require sorting. In that case I assume ORing vs ANDing won't help since all of the terms must be visited in order to sort them. However, when I ran the test query it included sorting and was still pretty fast so hopefully this won't be an issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:21pm UTC](https://discuss.elastic.co/t/very-large-terms-query/39838/6 "2017-07-05T23:21:51Z")

</div>