# Treatment of special characters in elasticsearch

**URL:** <https://discuss.elastic.co/t/treatment-of-special-characters-in-elasticsearch/20897>\
**Category:** Elasticsearch\
**Created:** [November 22, 2014, 6:05pm UTC](https://discuss.elastic.co/t/treatment-of-special-characters-in-elasticsearch/20897 "2014-11-22T18:05:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![prachicsa](https://avatars.discourse-cdn.com/v4/letter/p/f4b2a3/32.png) [@prachicsa](https://discuss.elastic.co/u/prachicsa)\
**Post date:** [November 22, 2014, 6:05pm UTC](https://discuss.elastic.co/t/treatment-of-special-characters-in-elasticsearch/20897/1 "2014-11-22T18:05:09Z")

</div>

I use the following analyzer:

curl -XPUT '[http://localhost:9200/sample/](http://localhost:9200/sample/)' -d '  
{  
"settings" : {  
"index": {  
"analysis": {  
"analyzer": {  
"default": {  
"type": "custom",  
"tokenizer": "keyword",  
"filter": ["trim", "lowercase"]}  
}  
}  
}  
}  
}'

Then when I try to insert some documents which contain special characters  
like % and etc, it converts in to hex.

1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8  
-\> actual value

1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8

-\> stored value.

Sample:

curl -XPUT '[http://localhost:9200/sample/strom/1](http://localhost:9200/sample/strom/1)' -d '{  
"user" : "user1",  
"message" : "1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8"  
}'

The problem started occurring only once the data crossed some million  
documents. Earlier it used store it as it is.

Now if I try to search using,

1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8

it is not able to retrieve the document. How do I deal with this? The  
behavior seems to non-deterministic in converting special character to hex.

I am unable to replicate the same issue on localmachine.

Can someone explain the mistake I am making?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [November 22, 2014, 8:45pm UTC](https://discuss.elastic.co/t/treatment-of-special-characters-in-elasticsearch/20897/2 "2014-11-22T20:45:22Z")

</div>

Check your client. If you use curl, and shell, it's the shell or curl that  
is handling characters, for example, URI percent encoding.

Elasticsearch, when it has received data, does not do any extra conversion,  
it expects UTF-8.

Jörg

On Sat, Nov 22, 2014 at 7:05 PM, [prachicsa@gmail.com](mailto:prachicsa@gmail.com) wrote:

> I use the following analyzer:
> 
> curl -XPUT '[http://localhost:9200/sample/](http://localhost:9200/sample/)' -d '  
> {  
> "settings" : {  
> "index": {  
> "analysis": {  
> "analyzer": {  
> "default": {  
> "type": "custom",  
> "tokenizer": "keyword",  
> "filter": ["trim", "lowercase"]}  
> }  
> }  
> }  
> }  
> }'
> 
> Then when I try to insert some documents which contain special characters  
> like % and etc, it converts in to hex.
> 
> 1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8  
> -\> actual value
> 
> 1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8
> 
> -\> stored value.
> 
> Sample:
> 
> curl -XPUT '[http://localhost:9200/sample/strom/1](http://localhost:9200/sample/strom/1)' -d '{  
> "user" : "user1",  
> "message" : "1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8"  
> }'
> 
> The problem started occurring only once the data crossed some million  
> documents. Earlier it used store it as it is.
> 
> Now if I try to search using,
> 
> 1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8
> 
> it is not able to retrieve the document. How do I deal with this? The  
> behavior seems to non-deterministic in converting special character to hex.
> 
> I am unable to replicate the same issue on localmachine.
> 
> Can someone explain the mistake I am making?
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHJE\_Ld2V5ya%2BcqVMg-vJi7FhLcJDO6w%3DoWch7obRiE1A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHJE_Ld2V5ya%2BcqVMg-vJi7FhLcJDO6w%3DoWch7obRiE1A%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![prachicsa](https://avatars.discourse-cdn.com/v4/letter/p/f4b2a3/32.png) [@prachicsa](https://discuss.elastic.co/u/prachicsa)\
**Post date:** [November 23, 2014, 3:27am UTC](https://discuss.elastic.co/t/treatment-of-special-characters-in-elasticsearch/20897/3 "2014-11-23T03:27:18Z")

</div>

I am using Java Transport client. Where do we have to specify about  
handling characters there?

On Sunday, November 23, 2014 2:15:52 AM UTC+5:30, Jörg Prante wrote:

> Check your client. If you use curl, and shell, it's the shell or curl that  
> is handling characters, for example, URI percent encoding.
> 
> Elasticsearch, when it has received data, does not do any extra  
> conversion, it expects UTF-8.
> 
> Jörg
> 
> On Sat, Nov 22, 2014 at 7:05 PM, \<[prac...@gmail.com](mailto:prac...@gmail.com) \<javascript:\>\> wrote:
> 
> > I use the following analyzer:
> > 
> > curl -XPUT '[http://localhost:9200/sample/](http://localhost:9200/sample/)' -d '  
> > {  
> > "settings" : {  
> > "index": {  
> > "analysis": {  
> > "analyzer": {  
> > "default": {  
> > "type": "custom",  
> > "tokenizer": "keyword",  
> > "filter": ["trim", "lowercase"]}  
> > }  
> > }  
> > }  
> > }  
> > }'
> > 
> > Then when I try to insert some documents which contain special characters  
> > like % and etc, it converts in to hex.
> > 
> > 1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8  
> > -\> actual value
> > 
> > 1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8
> > 
> > -\> stored value.
> > 
> > Sample:
> > 
> > curl -XPUT '[http://localhost:9200/sample/strom/1](http://localhost:9200/sample/strom/1)' -d '{  
> > "user" : "user1",  
> > "message" : "1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8"  
> > }'
> > 
> > The problem started occurring only once the data crossed some million  
> > documents. Earlier it used store it as it is.
> > 
> > Now if I try to search using,
> > 
> > 1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8
> > 
> > it is not able to retrieve the document. How do I deal with this? The  
> > behavior seems to non-deterministic in converting special character to hex.
> > 
> > I am unable to replicate the same issue on localmachine.
> > 
> > Can someone explain the mistake I am making?
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/32de1fd0-6caf-4bb6-b0aa-d37041a46384%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/32de1fd0-6caf-4bb6-b0aa-d37041a46384%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [November 23, 2014, 1:32pm UTC](https://discuss.elastic.co/t/treatment-of-special-characters-in-elasticsearch/20897/4 "2014-11-23T13:32:24Z")

</div>

Then you have an error in your program.

I tried it and it works. There is no character handling, except lowercase  
filter.

See:

> <https://gist.github.com/jprante/4e5cd1ac2220ef8c39ca>

Jörg

On Sun, Nov 23, 2014 at 4:27 AM, [prachicsa@gmail.com](mailto:prachicsa@gmail.com) wrote:

> I am using Java Transport client. Where do we have to specify about  
> handling characters there?
> 
> On Sunday, November 23, 2014 2:15:52 AM UTC+5:30, Jörg Prante wrote:
> 
> > Check your client. If you use curl, and shell, it's the shell or curl  
> > that is handling characters, for example, URI percent encoding.
> > 
> > Elasticsearch, when it has received data, does not do any extra  
> > conversion, it expects UTF-8.
> > 
> > Jörg
> > 
> > On Sat, Nov 22, 2014 at 7:05 PM, [prac...@gmail.com](mailto:prac...@gmail.com) wrote:
> > 
> > > I use the following analyzer:
> > > 
> > > curl -XPUT '[http://localhost:9200/sample/](http://localhost:9200/sample/)' -d '  
> > > {  
> > > "settings" : {  
> > > "index": {  
> > > "analysis": {  
> > > "analyzer": {  
> > > "default": {  
> > > "type": "custom",  
> > > "tokenizer": "keyword",  
> > > "filter": ["trim", "lowercase"]}  
> > > }  
> > > }  
> > > }  
> > > }  
> > > }'
> > > 
> > > Then when I try to insert some documents which contain special  
> > > characters like % and etc, it converts in to hex.
> > > 
> > > 1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIv  
> > > Y54iMiX\_149c95f02a8 -\> actual value
> > > 
> > > 1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8
> > > 
> > > -\> stored value.
> > > 
> > > Sample:
> > > 
> > > curl -XPUT '[http://localhost:9200/sample/strom/1](http://localhost:9200/sample/strom/1)' -d '{  
> > > "user" : "user1",  
> > > "message" : "1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8"  
> > > }'
> > > 
> > > The problem started occurring only once the data crossed some million  
> > > documents. Earlier it used store it as it is.
> > > 
> > > Now if I try to search using,
> > > 
> > > 1%2fPJJP3JV2C24iDfEu9XpHBaYxXh%2fdHTbmchB35SDznXO2g8Vz4D7GTIvY54iMiX\_149c95f02a8
> > > 
> > > it is not able to retrieve the document. How do I deal with this? The  
> > > behavior seems to non-deterministic in converting special character to hex.
> > > 
> > > I am unable to replicate the same issue on localmachine.
> > > 
> > > Can someone explain the mistake I am making?
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%  
> > > [40googlegroups.com](http://40googlegroups.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6c8dff11-8ab4-4acf-8e85-4b4c93b270f7%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/32de1fd0-6caf-4bb6-b0aa-d37041a46384%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/32de1fd0-6caf-4bb6-b0aa-d37041a46384%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/32de1fd0-6caf-4bb6-b0aa-d37041a46384%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/32de1fd0-6caf-4bb6-b0aa-d37041a46384%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFc6FBDVL-nU2wojLGdV0%2Bzrg4Pd%3DMT5x48X%2B-nhsuZhw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFc6FBDVL-nU2wojLGdV0%2Bzrg4Pd%3DMT5x48X%2B-nhsuZhw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:48am UTC](https://discuss.elastic.co/t/treatment-of-special-characters-in-elasticsearch/20897/5 "2017-07-06T00:48:13Z")

</div>