# Searching on special characters **URL:** \ **Category:** Elasticsearch\ **Created:** [January 28, 2013, 12:11pm UTC](https://discuss.elastic.co/t/searching-on-special-characters/10522 "2013-01-28T12:11:21Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![andrea\_crotti](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@andrea\_crotti](https://discuss.elastic.co/u/andrea_crotti)\ **Post date:** [January 28, 2013, 12:11pm UTC](https://discuss.elastic.co/t/searching-on-special-characters/10522/1 "2013-01-28T12:11:21Z")
I'm a bit lost with a couple of ES string issues. So the idea is that I receive a search string, and I would like to be able to search also on special characters. I've found out that ES strips out all the special characters, and I found that this should work for example: # need to add a special mapping to make this work NOT\_ANALYZE\_MAP = { "properties": { "description": { "type": "string", "index": "not\_analyzed" } } } CON.put\_mapping(DOC\_TYPE, NOT\_ANALYZE\_MAP, indices=[TEST\_INDEX]) But then suppose I index something like this: {'name': 'special', 'created\_datetime': '2012-02-14T13:10:47Z', 'doc\_type': 'User', 'ideas': range(20), 'description': 'Another tester with @special'}, pyes.StringQuery('@special', search\_fields=['description']) never returns anything (even with the mapping), but this: pyes.StringQuery('@special') always works, even if I think it should not without the mapping. Any idea of what could that be and how do I correctly allow special characters? Is there a quoting function somewhere to make me quote the special characters that would be interpreted in some other ways? -- ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [January 28, 2013, 6:48pm UTC](https://discuss.elastic.co/t/searching-on-special-characters/10522/2 "2013-01-28T18:48:07Z")
The latter search is probably working because it is defaulting to the \_all field, which is using the standard analyzer. A non-analyzed field will not work, since the field will be indexed as one term: "Another tester with @special". You need to create your own custom analyzer that does not strip out the special characters. Are you referring to the @ symbol? The issue with the standard analyzer is not that is strips out characters (char filter), but the tokenizer has specific word boundaries. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. You can see how a sentence is parsed by using the analysis API. You have a few options for tokenziers. You can use these whitespace analyzer, which might be too lenient for most case, or you can use the pattern tokenizer. There are many other tokenizers, but these should help with your case. All these concepts come from Lucene, so you can learn more about them by reading up on Lucene. Cheers, Ivan On Mon, Jan 28, 2013 at 4:11 AM, andrea crotti [andrea.crotti.0@gmail.com](mailto:andrea.crotti.0@gmail.com)wrote: > I'm a bit lost with a couple of ES string issues. > So the idea is that I receive a search string, and I would like to be able > to search also on special characters. > > I've found out that ES strips out all the special characters, and I found > that this should work for example: > > # need to add a special mapping to make this work > > NOT\_ANALYZE\_MAP = { > "properties": { > "description": { > "type": "string", > "index": "not\_analyzed" > } > } > } > CON.put\_mapping(DOC\_TYPE, NOT\_ANALYZE\_MAP, indices=[TEST\_INDEX]) > > But then suppose I index something like this: > {'name': 'special', > 'created\_datetime': '2012-02-14T13:10:47Z', > 'doc\_type': 'User', > 'ideas': range(20), > 'description': 'Another tester with @special'}, > > pyes.StringQuery('@special', search\_fields=['description']) > never returns anything (even with the mapping), but this: > > pyes.StringQuery('@special') > always works, even if I think it should not without the mapping. > > Any idea of what could that be and how do I correctly allow special > characters? Is there a quoting function somewhere to make me quote > the special characters that would be interpreted in some other ways? > > -- -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group, send email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![andrea\_crotti](https://avatars.discourse-cdn.com/v4/letter/a/73ab20/32.png) [@andrea\_crotti](https://discuss.elastic.co/u/andrea_crotti)\ **Post date:** [January 29, 2013, 10:41am UTC](https://discuss.elastic.co/t/searching-on-special-characters/10522/3 "2013-01-29T10:41:36Z")
Thanks for the reply.. I think it's not worth then after all, also because I then have to make sure I quote the special symbols that would be interpreted by ES, and so on.. On Monday, January 28, 2013 6:48:07 PM UTC, Ivan Brusic wrote: > The latter search is probably working because it is defaulting to the \_all > field, which is using the standard analyzer. A non-analyzed field will not > work, since the field will be indexed as one term: "Another tester with > @special". > > You need to create your own custom analyzer that does not strip out the > special characters. Are you referring to the @ symbol? The issue with the > standard analyzer is not that is strips out characters (char filter), but > the tokenizer has specific word boundaries. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/standard-analyzer.html) > > You can see how a sentence is parsed by using the analysis API. You have a > few options for tokenziers. You can use these whitespace analyzer, which > might be too lenient for most case, or you can use the pattern tokenizer. > There are many other tokenizers, but these should help with your case. All > these concepts come from Lucene, so you can learn more about them by > reading up on Lucene. > > Cheers, > > Ivan > > On Mon, Jan 28, 2013 at 4:11 AM, andrea crotti \<[andrea....@gmail.com](mailto:andrea....@gmail.com)\ > > > wrote: > > > I'm a bit lost with a couple of ES string issues. > > So the idea is that I receive a search string, and I would like to be > > able to search also on special characters. > > > > I've found out that ES strips out all the special characters, and I found > > that this should work for example: > > > > # need to add a special mapping to make this work > > > > NOT\_ANALYZE\_MAP = { > > "properties": { > > "description": { > > "type": "string", > > "index": "not\_analyzed" > > } > > } > > } > > CON.put\_mapping(DOC\_TYPE, NOT\_ANALYZE\_MAP, indices=[TEST\_INDEX]) > > > > But then suppose I index something like this: > > {'name': 'special', > > 'created\_datetime': '2012-02-14T13:10:47Z', > > 'doc\_type': 'User', > > 'ideas': range(20), > > 'description': 'Another tester with @special'}, > > > > pyes.StringQuery('@special', search\_fields=['description']) > > never returns anything (even with the mapping), but this: > > > > pyes.StringQuery('@special') > > always works, even if I think it should not without the mapping. > > > > Any idea of what could that be and how do I correctly allow special > > characters? Is there a quoting function somewhere to make me quote > > the special characters that would be interpreted in some other ways? > > > > -- -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:54am UTC](https://discuss.elastic.co/t/searching-on-special-characters/10522/4 "2017-07-06T02:54:13Z")