# Difficulties in searching in strings where words are separated by dots, underscores and hyphens **URL:** \ **Category:** Elasticsearch\ **Created:** [October 7, 2014, 7:28am UTC](https://discuss.elastic.co/t/difficulties-in-searching-in-strings-where-words-are-separated-by-dots-underscores-and-hyphens/20103 "2014-10-07T07:28:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Radim\_Bukovsky](https://avatars.discourse-cdn.com/v4/letter/r/f04885/32.png) [@Radim\_Bukovsky](https://discuss.elastic.co/u/Radim_Bukovsky)\ **Post date:** [October 7, 2014, 7:28am UTC](https://discuss.elastic.co/t/difficulties-in-searching-in-strings-where-words-are-separated-by-dots-underscores-and-hyphens/20103/1 "2014-10-07T07:28:02Z")
Hi, we use ElasticSearch in our file-sharing website. Our aim is to deliver the best search results as we can. We have now implemented ElasticSearch - Fuzzy Like This Query. But I am not convinced that is the best way for us. We have one search field for users. Keywords are searched in names and descriptions of uploaded files. Using Fuzzy Like This Query give us great results when description or filename (example: keyword is "trance", it finds "trance 2014", "Best trance Ever" etc.). Difficulties are with filenames/descriptions where the keywords are separated by dots (.), underscores (\_) or hyphen (-) (example: keyword is "trance", it does not find "Best.trance.Ever" or "Best-trance-Ever" or "Best\_trance\_Ever"). Do you have any advice for this? Is it possible to solve it within Fuzzy Like This Query? Thank you very much for your help. Best regards, Radim Bukovsky Settings (Fuzzy Like This Query): addFields(array('name', 'description')); setLikeText($file-\>name); setMaxQueryTerms(12); setMinSimilarity(1); -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9cdd5b1f-a728-44e5-9efb-c85251a00100%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9cdd5b1f-a728-44e5-9efb-c85251a00100%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [October 7, 2014, 7:41am UTC](https://discuss.elastic.co/t/difficulties-in-searching-in-strings-where-words-are-separated-by-dots-underscores-and-hyphens/20103/2 "2014-10-07T07:41:06Z")
Hi, this sound like an issue with analyzer to me. Can you share more info about analyzer configuration for your 'name' and 'description' fields? Regards, Lukas On Tue, Oct 7, 2014 at 9:28 AM, Radim Bukovský [rad.bukovsky@seznam.cz](mailto:rad.bukovsky@seznam.cz) wrote: > Hi, > we use Elasticsearch in our file-sharing website. Our aim is to deliver > the best search results as we can. We have now implemented Elasticsearch - > Fuzzy Like This Query. But I am not convinced that is the best way for us. > We have one search field for users. Keywords are searched in names and > descriptions of uploaded files. Using Fuzzy Like This Query give us great > results when description or filename (example: keyword is "trance", it > finds "trance 2014", "Best trance Ever" etc.). > Difficulties are with filenames/descriptions where the keywords are > separated by dots (.), underscores (\_) or hyphen (-) (example: keyword is > "trance", it does not find "Best.trance.Ever" or "Best-trance-Ever" or > "Best\_trance\_Ever"). Do you have any advice for this? > Is it possible to solve it within Fuzzy Like This Query? > > Thank you very much for your help. > > Best regards, > Radim Bukovsky > > Settings (Fuzzy Like This Query): > addFields(array('name', 'description')); > setLikeText($file-\>name); > setMaxQueryTerms(12); > setMinSimilarity(1); > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/9cdd5b1f-a728-44e5-9efb-c85251a00100%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9cdd5b1f-a728-44e5-9efb-c85251a00100%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/9cdd5b1f-a728-44e5-9efb-c85251a00100%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9cdd5b1f-a728-44e5-9efb-c85251a00100%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO9cvUbBiHQiHO7JEoPODOKVbaWo5M1MqMp95G8k%3DdubKyNnHA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO9cvUbBiHQiHO7JEoPODOKVbaWo5M1MqMp95G8k%3DdubKyNnHA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Radim\_Bukovsky](https://avatars.discourse-cdn.com/v4/letter/r/f04885/32.png) [@Radim\_Bukovsky](https://discuss.elastic.co/u/Radim_Bukovsky)\ **Post date:** [October 7, 2014, 2:24pm UTC](https://discuss.elastic.co/t/difficulties-in-searching-in-strings-where-words-are-separated-by-dots-underscores-and-hyphens/20103/3 "2014-10-07T14:24:03Z")
Hi, I am not sure if I understand what do you think. Could you be more specific? Many thanks. Settings: setFields(array('name^3', 'description')); ------------------\> ('fields') setQuery($search . ' extension:('.$filter.')'); -------------------\> ('query') setDefaultOperator('AND'); ------------------\> ('default\_operator') There is list of all parameters if helps: 'allow\_leading\_wildcard' ------------\> true 'enable\_position\_increments' -------\> true 'lowercase\_expanded\_terms' -------\> true 'fuzzy\_prefix\_length' ---------\> 0 'fuzzy\_min\_sim' ---------\> 0.5 'phrase\_slop' ---------\> 0 'boost' --------\> 1.0 'analyze\_wildcard' ----------\> true 'auto\_generate\_phrase\_queries' ------------ \> true 'use\_dis\_max' ----\> true 'tie\_breaker' ------\> 0 'rewrite' -----------\> "" Regards, Radim -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [October 7, 2014, 4:55pm UTC](https://discuss.elastic.co/t/difficulties-in-searching-in-strings-where-words-are-separated-by-dots-underscores-and-hyphens/20103/4 "2014-10-07T16:55:24Z")
Your issue is with the standard tokenizer, which will tokenize on most non word characters. Try using a whitespace or pattern tokenizer, which is dependent on your use case. -- Ivan On Oct 7, 2014 7:24 AM, "Radim Bukovský" [rad.bukovsky@seznam.cz](mailto:rad.bukovsky@seznam.cz) wrote: > Hi, > I am not sure if I understand what do you think. Could you be more > specific? Many thanks. > > Settings: > > setFields(array('name^3', 'description')); ------------------\> > ('fields') > setQuery($search . ' extension:('.$filter.')'); -------------------\> > ('query') > setDefaultOperator('AND'); ------------------\> > ('default\_operator') > > There is list of all parameters if helps: > > 'allow\_leading\_wildcard' ------------\> true > 'enable\_position\_increments' -------\> true > 'lowercase\_expanded\_terms' -------\> true > 'fuzzy\_prefix\_length' ---------\> 0 > 'fuzzy\_min\_sim' ---------\> 0.5 > 'phrase\_slop' ---------\> 0 > 'boost' --------\> 1.0 > 'analyze\_wildcard' ----------\> true > 'auto\_generate\_phrase\_queries' ------------ \> true > 'use\_dis\_max' ----\> true > 'tie\_breaker' ------\> 0 > 'rewrite' -----------\> "" > > Regards, > Radim > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp\_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [October 7, 2014, 6:36pm UTC](https://discuss.elastic.co/t/difficulties-in-searching-in-strings-where-words-are-separated-by-dots-underscores-and-hyphens/20103/5 "2014-10-07T18:36:48Z")
If you still need the standard analyzer's behavior for words but want to force separation on stuff containing dots and underscores you can use the mapper character filter ( [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-mapping-charfilter.html)) to convert those characters to spaces. Its pretty crude but it should work. You can also use the word delimiter token filter ( [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-word-delimiter-tokenfilter.html)) but it has more weird corner cases. Nik On Tue, Oct 7, 2014 at 12:55 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > Your issue is with the standard tokenizer, which will tokenize on most non > word characters. > > Try using a whitespace or pattern tokenizer, which is dependent on your > use case. > > -- > Ivan > On Oct 7, 2014 7:24 AM, "Radim Bukovský" [rad.bukovsky@seznam.cz](mailto:rad.bukovsky@seznam.cz) wrote: > > > Hi, > > I am not sure if I understand what do you think. Could you be more > > specific? Many thanks. > > > > Settings: > > > > setFields(array('name^3', 'description')); ------------------\> > > ('fields') > > setQuery($search . ' extension:('.$filter.')'); -------------------\> > > ('query') > > setDefaultOperator('AND'); ------------------\> > > ('default\_operator') > > > > There is list of all parameters if helps: > > > > 'allow\_leading\_wildcard' ------------\> true > > 'enable\_position\_increments' -------\> true > > 'lowercase\_expanded\_terms' -------\> true > > 'fuzzy\_prefix\_length' ---------\> 0 > > 'fuzzy\_min\_sim' ---------\> 0.5 > > 'phrase\_slop' ---------\> 0 > > 'boost' --------\> 1.0 > > 'analyze\_wildcard' ----------\> true > > 'auto\_generate\_phrase\_queries' ------------ \> true > > 'use\_dis\_max' ----\> true > > 'tie\_breaker' ------\> 0 > > 'rewrite' -----------\> "" > > > > Regards, > > Radim > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp\_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp\_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3-RBLstc0adGt9KyHzc%3DRWnAHP2LNN%2Bv-psbHDZAtO\_w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3-RBLstc0adGt9KyHzc%3DRWnAHP2LNN%2Bv-psbHDZAtO_w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Bryan\_Warner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bryan_warner/32/1225_2.png) [@Bryan\_Warner](https://discuss.elastic.co/u/Bryan_Warner)\ **Post date:** [October 8, 2014, 2:07pm UTC](https://discuss.elastic.co/t/difficulties-in-searching-in-strings-where-words-are-separated-by-dots-underscores-and-hyphens/20103/6 "2014-10-08T14:07:31Z")
To be able to customize the default word boundary properties in Lucene's StandardTokenizer, I created an Elasticsearch plugin to be able to do this - [GitHub - bbguitar77/elasticsearch-analysis-standardext](https://github.com/bbguitar77/elasticsearch-analysis-standardext) As mentioned before, there are other tokenizers / filters that can be used, but each one has its own drawbacks. Plus, you'll forgo all the benefits of the StandardTokenizer (e.g. dealing with invisible characters that aren't considered whitespace, being able to define conditional word break rules for punctuation symbols with respect to how they appear in the text). Hope this helps - Bryan On Tuesday, October 7, 2014 2:37:34 PM UTC-4, Nikolas Everett wrote: > If you still need the standard analyzer's behavior for words but want to > force separation on stuff containing dots and underscores you can use the > mapper character filter ( > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-mapping-charfilter.html)) > to convert those characters to spaces. Its pretty crude but it should > work. You can also use the word delimiter token filter ( > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-word-delimiter-tokenfilter.html)) > but it has more weird corner cases. > > Nik > > On Tue, Oct 7, 2014 at 12:55 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com) > \\> wrote: > > > Your issue is with the standard tokenizer, which will tokenize on most > > non word characters. > > > > Try using a whitespace or pattern tokenizer, which is dependent on your > > use case. > > > > -- > > Ivan > > On Oct 7, 2014 7:24 AM, "Radim Bukovský" \ > \\> wrote: > > > > > Hi, > > > I am not sure if I understand what do you think. Could you be more > > > specific? Many thanks. > > > > > > Settings: > > > > > > setFields(array('name^3', 'description')); ------------------\> > > > ('fields') > > > setQuery($search . ' extension:('.$filter.')'); > > > -------------------\> ('query') > > > setDefaultOperator('AND'); ------------------\> > > > ('default\_operator') > > > > > > There is list of all parameters if helps: > > > > > > 'allow\_leading\_wildcard' ------------\> true > > > 'enable\_position\_increments' -------\> true > > > 'lowercase\_expanded\_terms' -------\> true > > > 'fuzzy\_prefix\_length' ---------\> 0 > > > 'fuzzy\_min\_sim' ---------\> 0.5 > > > 'phrase\_slop' ---------\> 0 > > > 'boost' --------\> 1.0 > > > 'analyze\_wildcard' ----------\> true > > > 'auto\_generate\_phrase\_queries' ------------ \> true > > > 'use\_dis\_max' ----\> true > > > 'tie\_breaker' ------\> 0 > > > 'rewrite' -----------\> "" > > > > > > Regards, > > > Radim > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0c3f9aff-2971-49c2-afc5-d1054d1068fe%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp\_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp\_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCU-%3DxwRxsdHREWh59kp_wJhdHbRosK8Co9njLhgR%2BZbQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f4c26ea4-928e-4079-b170-c11a9bf38028%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f4c26ea4-928e-4079-b170-c11a9bf38028%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:57am UTC](https://discuss.elastic.co/t/difficulties-in-searching-in-strings-where-words-are-separated-by-dots-underscores-and-hyphens/20103/7 "2017-07-06T00:57:38Z")