# Surprised by deprecation of common\_terms query. What about its relevance features? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 16, 2021, 7:10pm UTC](https://discuss.elastic.co/t/surprised-by-deprecation-of-common-terms-query-what-about-its-relevance-features/270416 "2021-04-16T19:10:40Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![softwaredoug](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/softwaredoug/32/22681_2.png) [@softwaredoug](https://discuss.elastic.co/u/softwaredoug)\ **Post date:** [April 16, 2021, 7:10pm UTC](https://discuss.elastic.co/t/surprised-by-deprecation-of-common-terms-query-what-about-its-relevance-features/270416/1 "2021-04-16T19:10:40Z")
Hi all, I was actually excited to dig into an experiment at work using the [common terms](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-common-terms-query.html) query. I see however, it's been deprecated as the performance benefits are seen now in the match query, as [mentioned in this issue](https://github.com/olivere/elastic/issues/1198). However, this query provides relevance functionality I don't _think_ I can get in a match query (please correct me if I'm wrong). And the reason for deprecation seems to relate to max score / block WAND. I've never understood `common_terms` to be about speed, but more about relevance. I was surprised this didn't come up in the deprecation discussions. Specifically, what I like about it is to be able to set a cutoff frequency, and make low or high value terms mandatory (default AND) or not mandatory (default OR) based on document frequency. Am I missing where I can do this with other functionality now? ---
**Author:** ![joshdevins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshdevins/32/62865_2.png) [@joshdevins](https://discuss.elastic.co/u/joshdevins)\ **Post date:** [April 19, 2021, 10:03am UTC](https://discuss.elastic.co/t/surprised-by-deprecation-of-common-terms-query-what-about-its-relevance-features/270416/2 "2021-04-19T10:03:14Z")
> [@softwaredoug](#): > > Specifically, what I like about it is to be able to set a cutoff frequency, and make low or high value terms mandatory (default AND) or not mandatory (default OR) based on document frequency. Do you think the same effect can be achieved by tuning the BM25 parameters `k1` and `b`? With or without `minimum_should_match` tuning? Paging Dr. @jimczi for more advice and perhaps historical context. ---
**Author:** ![softwaredoug](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/softwaredoug/32/22681_2.png) [@softwaredoug](https://discuss.elastic.co/u/softwaredoug)\ **Post date:** [April 19, 2021, 1:00pm UTC](https://discuss.elastic.co/t/surprised-by-deprecation-of-common-terms-query-what-about-its-relevance-features/270416/3 "2021-04-19T13:00:31Z")
Thanks @joshdevins -\> yeah optimizing BM25 and min-should-match params have been a big focus of our work. However, with min should match, we can't choose _which_ tokens to remove. It just provides a hard floor on number of tokens. For example in a product search, if someone searches for "blue suede jacket" I'd prefer if "blue" was made optional (presumably high DF) but "suede" and "jacket" mandatory. So for the next round of experimentation, I had hoped to turn to `common_terms` as opposed to maintaining or computing a specific list of these low value terms outside the search engine. ---
**Author:** ![jimczi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jimczi/32/47985_2.png) [@jimczi](https://discuss.elastic.co/u/jimczi)\ **Post date:** [April 19, 2021, 5:17pm UTC](https://discuss.elastic.co/t/surprised-by-deprecation-of-common-terms-query-what-about-its-relevance-features/270416/4 "2021-04-19T17:17:44Z")
I always looked at this from a different angle. The question is not whether a term should be mandatory or not but rather how much it contributes to the overall score. The dance with minimum\_should\_match, common\_terms and all these advanced options is there to minimize the impact of always searching all terms. So when WAND was introduced, I thought that it could be a chance to simplify things further. Users can opt for the default behavior of considering all terms optional and at the same time rely on internal optimizations to ensure that we'll not consider all documents eagerly. I always found minimum\_should\_match and common\_terms difficult to approach. Finding the right configuration is tricky and whatever you find must be updated as the data and queries evolve. So that's a lot of burden for users that "just" want search to surface the most relevant results automatically. ---
**Author:** ![softwaredoug](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/softwaredoug/32/22681_2.png) [@softwaredoug](https://discuss.elastic.co/u/softwaredoug)\ **Post date:** [April 20, 2021, 2:21pm UTC](https://discuss.elastic.co/t/surprised-by-deprecation-of-common-terms-query-what-about-its-relevance-features/270416/5 "2021-04-20T14:21:00Z")
Thanks @jimczi - appreciate the response The specific case I'm working on is if you search for 'blue suede jacket', and let's say there's only 2 exact matches, you'll see 'blue AND suede AND jacket' matches. If below that two, there's spurious matches on blue or suede, then even above the fold, you'll see some irrelevant results. I was hoping to use common\_terms to be a little smarter about making some terms mandatory (like jacket) and others optional (blue, suede, perhaps), so some of these lower down results would drop off and not be shown at all. Of course if there's other ideas on how to tamp down the recall a bit, I'd be open to it... I know I could reissue the query, relaxing it strategically, or do a bit more before the query hits Elasticsearch. But I don't have access to doc freq and other useful index stats in a search service which would be useful in making this decision. Hope that context makes sense. Anyway, it's up to you guys whether you want to deprecate it, I just wanted to share perhaps one use case where common\_terms seems to help with relevance! Best! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 18, 2021, 2:21pm UTC](https://discuss.elastic.co/t/surprised-by-deprecation-of-common-terms-query-what-about-its-relevance-features/270416/6 "2021-05-18T14:21:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.