# Extracting fuzzy match terms **URL:** \ **Category:** Elasticsearch\ **Created:** [April 27, 2015, 4:08pm UTC](https://discuss.elastic.co/t/extracting-fuzzy-match-terms/23424 "2015-04-27T16:08:55Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Graham\_Turner](https://avatars.discourse-cdn.com/v4/letter/g/c68b51/32.png) [@Graham\_Turner](https://discuss.elastic.co/u/Graham_Turner)\ **Post date:** [April 27, 2015, 4:08pm UTC](https://discuss.elastic.co/t/extracting-fuzzy-match-terms/23424/1 "2015-04-27T16:08:55Z")
Hi, I'm working on a proof-of-concept for a client, replacing an existing legacy search system with an elastic based alternative. One of the requirements that comes from the existing system is that, when performing a fuzzy or wildcard search, the user can view all the matching terms, and include/exclude them manually from the subsequent search. Thus, if a fuzzy search for 'graham' is submitted (or a wildcard like 'gr_m_'), it might match grayam, graeme, grahum, grahem, etc. The users want to be able to see this list of matched terms, then, for instance, exclude 'grayam' from the expanded terms list, so that all the other expansions are used, but not the specifically excluded one. I’m struggling to retrieve this list of terms in the first place. Ideally I’d like to submit a simple query for a fuzzy or wildcard term, and have it return just the possible matching terms (up to a given limit). I’ve had reasonable success using the term suggester for fuzzy-type responses, but can’t use this for wildcard expansions. Is there a good way to do this using 'out-of-the-box' elastic functionality? Any advice / hints gratefully accepted! Thanks Graham -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b41c54b-f749-4cf1-902a-f3d0ce145d29%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b41c54b-f749-4cf1-902a-f3d0ce145d29%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 27, 2015, 4:41pm UTC](https://discuss.elastic.co/t/extracting-fuzzy-match-terms/23424/2 "2015-04-27T16:41:17Z")
Hi Graham, If you were to use the highlighter functionality you would essentially "see what the search engine saw". With some client-side coding you could parse out the expanded search terms because they would be surrounded by tags in matching docs. Of course this wouldn't provide a de-duped list of terms and would be inefficient to return an exhaustive list of all expansions used but may be an approach to investigate. Cheers Mark On Monday, April 27, 2015 at 5:08:55 PM UTC+1, Graham Turner wrote: > Hi, > > I'm working on a proof-of-concept for a client, replacing an existing > legacy search system with an elastic based alternative. One of the > requirements that comes from the existing system is that, when performing a > fuzzy or wildcard search, the user can view all the matching terms, and > include/exclude them manually from the subsequent search. > > Thus, if a fuzzy search for 'graham' is submitted (or a wildcard like > 'gr_m_'), it might match grayam, graeme, grahum, grahem, etc. The users > want to be able to see this list of matched terms, then, for instance, > exclude 'grayam' from the expanded terms list, so that all the other > expansions are used, but not the specifically excluded one. > > I’m struggling to retrieve this list of terms in the first place. Ideally > I’d like to submit a simple query for a fuzzy or wildcard term, and have it > return just the possible matching terms (up to a given limit). > > I’ve had reasonable success using the term suggester for fuzzy-type > responses, but can’t use this for wildcard expansions. > > Is there a good way to do this using 'out-of-the-box' elastic > functionality? > > Any advice / hints gratefully accepted! > > Thanks > > Graham -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6ef4ede0-78d4-4aa2-82d6-4041cadce89d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ef4ede0-78d4-4aa2-82d6-4041cadce89d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Graham\_Turner](https://avatars.discourse-cdn.com/v4/letter/g/c68b51/32.png) [@Graham\_Turner](https://discuss.elastic.co/u/Graham_Turner)\ **Post date:** [April 28, 2015, 11:00am UTC](https://discuss.elastic.co/t/extracting-fuzzy-match-terms/23424/3 "2015-04-28T11:00:49Z")
Thanks Mark. I did wonder about the highlighter, but using it would mean potentially retrieving every hit and parsing it, which feels pretty impractical for large searches. Presumably the fuzzy query has to identify a full list of matching terms internally - is there any way we could somehow hook into this, or retrieve the list separately to the query results? A mechanism similar to the suggester, just accepting a single fuzzy term or a wildcard term would be perfect. I appreciate this probably isn't a common request, but I'm sure it would have other use cases. Something to consider for a future release perhaps? 🙂 Cheers Graham On Monday, 27 April 2015 17:41:17 UTC+1, [ma...@elastic.co](mailto:ma...@elastic.co) wrote: > Hi Graham, > If you were to use the highlighter functionality you would essentially > "see what the search engine saw". > With some client-side coding you could parse out the expanded search terms > because they would be surrounded by tags in matching docs. > Of course this wouldn't provide a de-duped list of terms and would be > inefficient to return an exhaustive list of all expansions used but may be > an approach to investigate. > > Cheers > Mark > > On Monday, April 27, 2015 at 5:08:55 PM UTC+1, Graham Turner wrote: > > > Hi, > > > > I'm working on a proof-of-concept for a client, replacing an existing > > legacy search system with an elastic based alternative. One of the > > requirements that comes from the existing system is that, when performing a > > fuzzy or wildcard search, the user can view all the matching terms, and > > include/exclude them manually from the subsequent search. > > > > Thus, if a fuzzy search for 'graham' is submitted (or a wildcard like > > 'gr_m_'), it might match grayam, graeme, grahum, grahem, etc. The users > > want to be able to see this list of matched terms, then, for instance, > > exclude 'grayam' from the expanded terms list, so that all the other > > expansions are used, but not the specifically excluded one. > > > > I’m struggling to retrieve this list of terms in the first place. > > Ideally I’d like to submit a simple query for a fuzzy or wildcard term, and > > have it return just the possible matching terms (up to a given limit). > > > > I’ve had reasonable success using the term suggester for fuzzy-type > > responses, but can’t use this for wildcard expansions. > > > > Is there a good way to do this using 'out-of-the-box' elastic > > functionality? > > > > Any advice / hints gratefully accepted! > > > > Thanks > > > > Graham -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bb544adc-bf72-4d9c-a000-2ce08604488c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bb544adc-bf72-4d9c-a000-2ce08604488c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 28, 2015, 8:26pm UTC](https://discuss.elastic.co/t/extracting-fuzzy-match-terms/23424/4 "2015-04-28T20:26:12Z")
All Lucene queries implement extractTerms [1] and this API is used by highlighter implementations to get the expanded set of terms in wildcards/fuzzy etc. This set of terms isn't exposed directly in elasticsearch today but you may be able to hack something together using scripts or a custom Java plugin - look at SearchContext.current().query().extractTerms(). Cheers Mark [1] [Query (Lucene 5.1.0 API)](http://lucene.apache.org/core/5_1_0/core/org/apache/lucene/search/Query.html#extractTerms(java.util.Set)) On Tuesday, April 28, 2015 at 12:00:49 PM UTC+1, Graham Turner wrote: > Thanks Mark. > > I did wonder about the highlighter, but using it would mean potentially > retrieving every hit and parsing it, which feels pretty impractical for > large searches. > > Presumably the fuzzy query has to identify a full list of matching terms > internally - is there any way we could somehow hook into this, or retrieve > the list separately to the query results? A mechanism similar to the > suggester, just accepting a single fuzzy term or a wildcard term would be > perfect. I appreciate this probably isn't a common request, but I'm sure > it would have other use cases. Something to consider for a future release > perhaps? 🙂 > > Cheers > > Graham > > On Monday, 27 April 2015 17:41:17 UTC+1, [ma...@elastic.co](mailto:ma...@elastic.co) wrote: > > > Hi Graham, > > If you were to use the highlighter functionality you would essentially > > "see what the search engine saw". > > With some client-side coding you could parse out the expanded search > > terms because they would be surrounded by tags in matching docs. > > Of course this wouldn't provide a de-duped list of terms and would be > > inefficient to return an exhaustive list of all expansions used but may be > > an approach to investigate. > > > > Cheers > > Mark > > > > On Monday, April 27, 2015 at 5:08:55 PM UTC+1, Graham Turner wrote: > > > > > Hi, > > > > > > I'm working on a proof-of-concept for a client, replacing an existing > > > legacy search system with an elastic based alternative. One of the > > > requirements that comes from the existing system is that, when performing a > > > fuzzy or wildcard search, the user can view all the matching terms, and > > > include/exclude them manually from the subsequent search. > > > > > > Thus, if a fuzzy search for 'graham' is submitted (or a wildcard like > > > 'gr_m_'), it might match grayam, graeme, grahum, grahem, etc. The users > > > want to be able to see this list of matched terms, then, for instance, > > > exclude 'grayam' from the expanded terms list, so that all the other > > > expansions are used, but not the specifically excluded one. > > > > > > I’m struggling to retrieve this list of terms in the first place. > > > Ideally I’d like to submit a simple query for a fuzzy or wildcard term, and > > > have it return just the possible matching terms (up to a given limit). > > > > > > I’ve had reasonable success using the term suggester for fuzzy-type > > > responses, but can’t use this for wildcard expansions. > > > > > > Is there a good way to do this using 'out-of-the-box' elastic > > > functionality? > > > > > > Any advice / hints gratefully accepted! > > > > > > Thanks > > > > > > Graham -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d8672e94-9063-4005-9d53-15b5cd0c6beb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d8672e94-9063-4005-9d53-15b5cd0c6beb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Graham\_Turner](https://avatars.discourse-cdn.com/v4/letter/g/c68b51/32.png) [@Graham\_Turner](https://discuss.elastic.co/u/Graham_Turner)\ **Post date:** [April 29, 2015, 1:39pm UTC](https://discuss.elastic.co/t/extracting-fuzzy-match-terms/23424/5 "2015-04-29T13:39:24Z")
That sounds interesting - I'll have a look and see if we can pull something together Cheers! On Tuesday, 28 April 2015 21:26:12 UTC+1, [ma...@elastic.co](mailto:ma...@elastic.co) wrote: > All Lucene queries implement extractTerms [1] and this API is used by > highlighter implementations to get the expanded set of terms in > wildcards/fuzzy etc. > This set of terms isn't exposed directly in elasticsearch today but you > may be able to hack something together using scripts or a custom Java > plugin - look at SearchContext.current().query().extractTerms(). > > Cheers > Mark > > [1] > [Query (Lucene 5.1.0 API)](http://lucene.apache.org/core/5_1_0/core/org/apache/lucene/search/Query.html#extractTerms(java.util.Set)) > > On Tuesday, April 28, 2015 at 12:00:49 PM UTC+1, Graham Turner wrote: > > > Thanks Mark. > > > > I did wonder about the highlighter, but using it would mean potentially > > retrieving every hit and parsing it, which feels pretty impractical for > > large searches. > > > > Presumably the fuzzy query has to identify a full list of matching terms > > internally - is there any way we could somehow hook into this, or retrieve > > the list separately to the query results? A mechanism similar to the > > suggester, just accepting a single fuzzy term or a wildcard term would be > > perfect. I appreciate this probably isn't a common request, but I'm sure > > it would have other use cases. Something to consider for a future release > > perhaps? 🙂 > > > > Cheers > > > > Graham > > > > On Monday, 27 April 2015 17:41:17 UTC+1, [ma...@elastic.co](mailto:ma...@elastic.co) wrote: > > > > > Hi Graham, > > > If you were to use the highlighter functionality you would essentially > > > "see what the search engine saw". > > > With some client-side coding you could parse out the expanded search > > > terms because they would be surrounded by tags in matching docs. > > > Of course this wouldn't provide a de-duped list of terms and would be > > > inefficient to return an exhaustive list of all expansions used but may be > > > an approach to investigate. > > > > > > Cheers > > > Mark > > > > > > On Monday, April 27, 2015 at 5:08:55 PM UTC+1, Graham Turner wrote: > > > > > > > Hi, > > > > > > > > I'm working on a proof-of-concept for a client, replacing an existing > > > > legacy search system with an elastic based alternative. One of the > > > > requirements that comes from the existing system is that, when performing a > > > > fuzzy or wildcard search, the user can view all the matching terms, and > > > > include/exclude them manually from the subsequent search. > > > > > > > > Thus, if a fuzzy search for 'graham' is submitted (or a wildcard like > > > > 'gr_m_'), it might match grayam, graeme, grahum, grahem, etc. The users > > > > want to be able to see this list of matched terms, then, for instance, > > > > exclude 'grayam' from the expanded terms list, so that all the other > > > > expansions are used, but not the specifically excluded one. > > > > > > > > I’m struggling to retrieve this list of terms in the first place. > > > > Ideally I’d like to submit a simple query for a fuzzy or wildcard term, and > > > > have it return just the possible matching terms (up to a given limit). > > > > > > > > I’ve had reasonable success using the term suggester for fuzzy-type > > > > responses, but can’t use this for wildcard expansions. > > > > > > > > Is there a good way to do this using 'out-of-the-box' elastic > > > > functionality? > > > > > > > > Any advice / hints gratefully accepted! > > > > > > > > Thanks > > > > > > > > Graham -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/29b83988-524f-47ff-bb3d-93f6685f58f3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/29b83988-524f-47ff-bb3d-93f6685f58f3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:17am UTC](https://discuss.elastic.co/t/extracting-fuzzy-match-terms/23424/6 "2017-07-06T00:17:03Z")