# What to do with non utf8 characters
**URL:** https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163
**Category:** Elasticsearch
**Created:** [December 15, 2011, 6:49am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163 "2011-12-15T06:49:16Z")
**Posts on this page:** 19
**Page:** 1
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 15, 2011, 6:49am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/1 "2011-12-15T06:49:16Z")
Hi ,
I am taking some text from the web and giving it to elasticSearch.
my problem is that , there are so many non utf8 characters in the text lile
�,“ã etc.
Now i used java's utf8 convertion thing but its replacing all such
characters with question mark.
How can i convert the text in such a way that , the text can be insrted
into elasticSearch and none of the
special characters are not lost.
Thanks
Vineeth
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 15, 2011, 7:15am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/2 "2011-12-15T07:15:23Z")
Between , this is the code i used to convert to utf8
contant -\> s
bbuf = encoder.encode(CharBuffer.wrap(content));
// Convert ISO-LATIN-1 bytes in a ByteBuffer to a character
ByteBuffer and then to a string.
// The new ByteBuffer is ready to be read.
CharBuffer cbuf = decoder.decode(bbuf);
s = cbuf.toString();
byte bytes = s.getBytes("ISO-8859-1");
s = new String(bytes, "UTF-8");
On Thu, Dec 15, 2011 at 12:19 PM, Vineeth Mohan
[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> Hi ,
>
> I am taking some text from the web and giving it to elasticSearch.
> my problem is that , there are so many non utf8 characters in the text
> lile �,“ã etc.
> Now i used java's utf8 convertion thing but its replacing all such
> characters with question mark.
>
> How can i convert the text in such a way that , the text can be insrted
> into elasticSearch and none of the
> special characters are not lost.
>
> Thanks
> Vineeth
---
### Author:  [@Karussell1](https://discuss.elastic.co/u/Karussell1)
#### Post date: [December 15, 2011, 8:28am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/3 "2011-12-15T08:28:10Z")
> its replacing all such characters with question mark.
That is a sign that your text is not UTF8. Make sure your source
documents are what you guessed (latin1?) and that they get properly
converted to UTF8. (I do not understand your converting code. Try to
explain what you want :))
Peter.
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 15, 2011, 9:50am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/4 "2011-12-15T09:50:19Z")
What is the general encoding of documents in internet , is it ISO-8859-1 or
something else.
Thanks
Vineeth
On Thu, Dec 15, 2011 at 1:58 PM, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)wrote:
> > its replacing all such characters with question mark.
>
> That is a sign that your text is not UTF8. Make sure your source
> documents are what you guessed (latin1?) and that they get properly
> converted to UTF8. (I do not understand your converting code. Try to
> explain what you want :))
>
> Peter.
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 15, 2011, 10:36am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/5 "2011-12-15T10:36:51Z")
When i tries to give leasticSearch following text -
"Small businesses potentially in line for a lighter reporting load include
those with an annual turnover of less than £440,000, net assets of less
than £220,000 and fewer than ten employees"
1. Through my java application - Basically my java application takes
this info from a webpage , and gives it to elasticSearch. ES complaints it
cant understand £ and it fails.
2. After filtering through below code -
byte bytes = s.getBytes("ISO-8859-1");
s = new String(bytes, "UTF-8");
Here £ is converted to �
3. I copy it to a file in my home using bash and it goes it fine.
I am really getting worked out stressed out solving this issue from my java
application side.
Any pointers will help.
Thanks
Vineeth
On Thu, Dec 15, 2011 at 3:20 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> What is the general encoding of documents in internet , is it ISO-8859-1
> or something else.
>
> Thanks
> Vineeth
>
> On Thu, Dec 15, 2011 at 1:58 PM, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)wrote:
>
> > > its replacing all such characters with question mark.
> >
> > That is a sign that your text is not UTF8. Make sure your source
> > documents are what you guessed (latin1?) and that they get properly
> > converted to UTF8. (I do not understand your converting code. Try to
> > explain what you want :))
> >
> > Peter.
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 16, 2011, 3:46am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/6 "2011-12-16T03:46:38Z")
Hi All,
My team is stuck on this. It would be of great if someone can help.
Thanks
Vineeth
On Thu, Dec 15, 2011 at 4:06 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> When i tries to give leasticSearch following text -
> "Small businesses potentially in line for a lighter reporting load include
> those with an annual turnover of less than £440,000, net assets of less
> than £220,000 and fewer than ten employees"
>
> 1. Through my java application - Basically my java application takes
> this info from a webpage , and gives it to elasticSearch. ES complaints it
> cant understand £ and it fails.
> 2. After filtering through below code -
> byte bytes = s.getBytes("ISO-8859-1");
> s = new String(bytes, "UTF-8");
> Here £ is converted to �
> 3. I copy it to a file in my home using bash and it goes it fine.
>
> I am really getting worked out stressed out solving this issue from my
> java application side.
> Any pointers will help.
>
> Thanks
> Vineeth
>
> On Thu, Dec 15, 2011 at 3:20 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
>
> > What is the general encoding of documents in internet , is it ISO-8859-1
> > or something else.
> >
> > Thanks
> > Vineeth
> >
> > On Thu, Dec 15, 2011 at 1:58 PM, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)wrote:
> >
> > > > its replacing all such characters with question mark.
> > >
> > > That is a sign that your text is not UTF8. Make sure your source
> > > documents are what you guessed (latin1?) and that they get properly
> > > converted to UTF8. (I do not understand your converting code. Try to
> > > explain what you want :))
> > >
> > > Peter.
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 16, 2011, 7:04am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/7 "2011-12-16T07:04:26Z")
Finally came up with the fix
```
CharsetDetector cs=new CharsetDetector();
cs.setText(content.getBytes());
String sourceEncoding=cs.detect().getName();
logger.debug("ENCODING is"+sourceEncoding);
if(sourceEncoding.equals("Big5")){
logger.debug("ENCO skipping as encoding is Big5 , content is
```
"+content);
return content;
}
String s =null ;
try {
byte bytes = content.getBytes(sourceEncoding);
s = new String(bytes, "UTF-8");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
Thanks
Vineeth
On Fri, Dec 16, 2011 at 9:16 AM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> Hi All,
>
> My team is stuck on this. It would be of great if someone can help.
>
> Thanks
> Vineeth
>
> On Thu, Dec 15, 2011 at 4:06 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
>
> > When i tries to give leasticSearch following text -
> > "Small businesses potentially in line for a lighter reporting load
> > include those with an annual turnover of less than £440,000, net assets of
> > less than £220,000 and fewer than ten employees"
> >
> > 1. Through my java application - Basically my java application takes
> > this info from a webpage , and gives it to elasticSearch. ES complaints it
> > cant understand £ and it fails.
> > 2. After filtering through below code -
> > byte bytes = s.getBytes("ISO-8859-1");
> > s = new String(bytes, "UTF-8");
> > Here £ is converted to �
> > 3. I copy it to a file in my home using bash and it goes it fine.
> >
> > I am really getting worked out stressed out solving this issue from my
> > java application side.
> > Any pointers will help.
> >
> > Thanks
> > Vineeth
> >
> > On Thu, Dec 15, 2011 at 3:20 PM, Vineeth Mohan \<[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)
> >
> > > wrote:
> >
> > > What is the general encoding of documents in internet , is it ISO-8859-1
> > > or something else.
> > >
> > > Thanks
> > > Vineeth
> > >
> > > On Thu, Dec 15, 2011 at 1:58 PM, Karussell \<[tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)
> > >
> > > > wrote:
> > >
> > > > > its replacing all such characters with question mark.
> > > >
> > > > That is a sign that your text is not UTF8. Make sure your source
> > > > documents are what you guessed (latin1?) and that they get properly
> > > > converted to UTF8. (I do not understand your converting code. Try to
> > > > explain what you want :))
> > > >
> > > > Peter.
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 16, 2011, 7:40am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/8 "2011-12-16T07:40:56Z")
Even that is not working ... am giving up ☹
~Vineeth
On Fri, Dec 16, 2011 at 12:34 PM, Vineeth Mohan
[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> Finally came up with the fix
>
> ```
> CharsetDetector cs=new CharsetDetector();
> cs.setText(content.getBytes());
> String sourceEncoding=cs.detect().getName();
> logger.debug("ENCODING is"+sourceEncoding);
> if(sourceEncoding.equals("Big5")){
> logger.debug("ENCO skipping as encoding is Big5 , content is
>
> ```
>
> "+content);
> return content;
> }
> String s =null ;
> try {
> byte bytes = content.getBytes(sourceEncoding);
>
> ```
> s = new String(bytes, "UTF-8");
> } catch (UnsupportedEncodingException e) {
> e.printStackTrace();
> }
>
> ```
>
> Thanks
> Vineeth
>
> On Fri, Dec 16, 2011 at 9:16 AM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
>
> > Hi All,
> >
> > My team is stuck on this. It would be of great if someone can help.
> >
> > Thanks
> > Vineeth
> >
> > On Thu, Dec 15, 2011 at 4:06 PM, Vineeth Mohan \<[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)
> >
> > > wrote:
> >
> > > When i tries to give leasticSearch following text -
> > > "Small businesses potentially in line for a lighter reporting load
> > > include those with an annual turnover of less than £440,000, net assets of
> > > less than £220,000 and fewer than ten employees"
> > >
> > > 1. Through my java application - Basically my java application takes
> > > this info from a webpage , and gives it to elasticSearch. ES complaints it
> > > cant understand £ and it fails.
> > > 2. After filtering through below code -
> > > byte bytes = s.getBytes("ISO-8859-1");
> > > s = new String(bytes, "UTF-8");
> > > Here £ is converted to �
> > > 3. I copy it to a file in my home using bash and it goes it fine.
> > >
> > > I am really getting worked out stressed out solving this issue from my
> > > java application side.
> > > Any pointers will help.
> > >
> > > Thanks
> > > Vineeth
> > >
> > > On Thu, Dec 15, 2011 at 3:20 PM, Vineeth Mohan \<
> > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote:
> > >
> > > > What is the general encoding of documents in internet , is it
> > > > ISO-8859-1 or something else.
> > > >
> > > > Thanks
> > > > Vineeth
> > > >
> > > > On Thu, Dec 15, 2011 at 1:58 PM, Karussell \<
> > > > [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)\> wrote:
> > > >
> > > > > > its replacing all such characters with question mark.
> > > > >
> > > > > That is a sign that your text is not UTF8. Make sure your source
> > > > > documents are what you guessed (latin1?) and that they get properly
> > > > > converted to UTF8. (I do not understand your converting code. Try to
> > > > > explain what you want :))
> > > > >
> > > > > Peter.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [December 16, 2011, 7:59am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/9 "2011-12-16T07:59:42Z")
Encoding is sometimes very complicated...
How your web pages encode form submission?I suggest you tell the user browser
that everything is UTF-8.
In a JSP, you can add this directive : \<%@ page contentType="text/html;
charset=UTF-8" %\>
Save your JSP file in UTF-8 (in case you have special chars in it).
I don't know if you can tell ES that you are using another encoding. It's
perhaps only a JVM parameter.
HTH
David.
Le 16 décembre 2011 à 08:40, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) a écrit :
> Even that is not working ... am giving up ☹
>
> ~Vineeth
>
> On Fri, Dec 16, 2011 at 12:34 PM, Vineeth Mohan
> [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
>
> > Finally came up with the fix
> >
> > ```
> > CharsetDetector cs=new CharsetDetector();
> > cs.setText(content.getBytes());
> > String sourceEncoding=cs.detect().getName();
> > logger.debug("ENCODING is"+sourceEncoding);
> > if(sourceEncoding.equals("Big5")){
> > logger.debug("ENCO skipping as encoding is Big5 , content is
> >
> > ```
> >
> > "+content);
> > return content;
> > }
> > String s =null ;
> > try {
> > byte bytes = content.getBytes(sourceEncoding);
> >
> > ```
> > s = new String(bytes, "UTF-8");
> > } catch (UnsupportedEncodingException e) {
> > e.printStackTrace();
> > }
> >
> > ```
> >
> > Thanks
> > Vineeth
> >
> > On Fri, Dec 16, 2011 at 9:16 AM, Vineeth Mohan
> > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> >
> > > Hi All,
> > >
> > > My team is stuck on this. It would be of great if someone can help.
> > >
> > > Thanks
> > > Vineeth
> > >
> > > On Thu, Dec 15, 2011 at 4:06 PM, Vineeth Mohan \<[vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)
> > >
> > > > wrote:
> > >
> > > > When i tries to give leasticSearch following text -
> > > > "Small businesses potentially in line for a lighter reporting load
> > > > include those with an annual turnover of less than £440,000, net assets of
> > > > less than £220,000 and fewer than ten employees"
> > > >
> > > > 1. Through my java application - Basically my java application takes
> > > > this info from a webpage , and gives it to elasticSearch. ES complaints
> > > > it
> > > > cant understand £ and it fails.
> > > > 2. After filtering through below code -
> > > > byte bytes = s.getBytes("ISO-8859-1");
> > > > s = new String(bytes, "UTF-8");
> > > > Here £ is converted to �
> > > > 3. I copy it to a file in my home using bash and it goes it fine.
> > > >
> > > > I am really getting worked out stressed out solving this issue from my
> > > > java application side.
> > > > Any pointers will help.
> > > >
> > > > Thanks
> > > > Vineeth
> > > >
> > > > On Thu, Dec 15, 2011 at 3:20 PM, Vineeth Mohan \<
> > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote:
> > > >
> > > > > What is the general encoding of documents in internet , is it
> > > > > ISO-8859-1 or something else.
> > > > >
> > > > > Thanks
> > > > > Vineeth
> > > > >
> > > > > On Thu, Dec 15, 2011 at 1:58 PM, Karussell \<
> > > > > [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)\> wrote:
> > > > >
> > > > > > > its replacing all such characters with question mark.
> > > > > >
> > > > > > That is a sign that your text is not UTF8. Make sure your source
> > > > > > documents are what you guessed (latin1?) and that they get properly
> > > > > > converted to UTF8. (I do not understand your converting code. Try to
> > > > > > explain what you want :))
> > > > > >
> > > > > > Peter.
--
David Pilato
[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
Twitter : @dadoonet
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 16, 2011, 8:15am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/10 "2011-12-16T08:15:43Z")
I am not using JSP or anything.
My application when given a web page link , goes there get the text in
there and gives that to elasticSearch.
now lets say the link given is -
[http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm\_source=feedburner&utm\_medium=feed&utm\_campaign=Feed%3A+fortunetermsheet+(Fortune+Finance%3A+Term+Sheet)](http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+fortunetermsheet+%28Fortune+Finance%3A+Term+Sheet%29)
It fails while i sent it tru my application.
Elasticsearch gives the following error - [gist:1485063 · GitHub](https://gist.github.com/1485063)
Now if i copy the text from that site manually , write it to a file from
bash shell or say gedit , and then run a normal curl command
to hit the elasticSearch to feed it data , things work very fine.
What should i do to achive the latter from my application.
Am behind this issue for couple of days.
Any help would be appreciated.
Thanks
Vineeth
On Fri, Dec 16, 2011 at 1:29 PM, [david@pilato.fr](mailto:david@pilato.fr) [david@pilato.fr](mailto:david@pilato.fr) wrote:
> \*\*
>
> Encoding is sometimes very complicated...
>
> How your web pages encode form submission? I suggest you tell the user
> browser that everything is UTF-8.
>
> In a JSP, you can add this directive : \<%@ page contentType="text/html;
> charset=UTF-8" %\>
>
> Save your JSP file in UTF-8 (in case you have special chars in it).
>
> I don't know if you can tell ES that you are using another encoding. It's
> perhaps only a JVM parameter.
>
> HTH
>
> David.
>
> Le 16 décembre 2011 à 08:40, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com) a
> écrit :
>
> > Even that is not working ... am giving up ☹
> >
> > ~Vineeth
> >
> > On Fri, Dec 16, 2011 at 12:34 PM, Vineeth Mohan
> > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> >
> > > Finally came up with the fix
> > >
> > > ```
> > > CharsetDetector cs=new CharsetDetector();
> > > cs.setText(content.getBytes());
> > > String sourceEncoding=cs.detect().getName();
> > > logger.debug("ENCODING is"+sourceEncoding);
> > > if(sourceEncoding.equals("Big5")){
> > > logger.debug("ENCO skipping as encoding is Big5 , content
> > >
> > > ```
>
> is
>
> > > "+content);
> > > return content;
> > > }
> > > String s =null ;
> > > try {
> > > byte bytes = content.getBytes(sourceEncoding);
> > >
> > > ```
> > > s = new String(bytes, "UTF-8");
> > > } catch (UnsupportedEncodingException e) {
> > > e.printStackTrace();
> > > }
> > >
> > > ```
> > >
> > > Thanks
> > > Vineeth
> > >
> > > On Fri, Dec 16, 2011 at 9:16 AM, Vineeth Mohan \<
> > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\>wrote:
> > >
> > > > Hi All,
> > > >
> > > > My team is stuck on this. It would be of great if someone can help.
> > > >
> > > > Thanks
> > > > Vineeth
> > > >
> > > > On Thu, Dec 15, 2011 at 4:06 PM, Vineeth Mohan \<
> > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)
> > > >
> > > > > wrote:
> > > >
> > > > > When i tries to give leasticSearch following text -
> > > > > "Small businesses potentially in line for a lighter reporting load
> > > > > include those with an annual turnover of less than £440,000, net
> > > > > assets of
> > > > > less than £220,000 and fewer than ten employees"
> > > > >
> > > > > 1. Through my java application - Basically my java application
> > > > > takes
> > > > > this info from a webpage , and gives it to elasticSearch. ES
> > > > > complaints it
> > > > > cant understand £ and it fails.
> > > > > 2. After filtering through below code -
> > > > > byte bytes = s.getBytes("ISO-8859-1");
> > > > > s = new String(bytes, "UTF-8");
> > > > > Here £ is converted to �
> > > > > 3. I copy it to a file in my home using bash and it goes it fine.
> > > > >
> > > > > I am really getting worked out stressed out solving this issue from
> > > > > my
> > > > > java application side.
> > > > > Any pointers will help.
> > > > >
> > > > > Thanks
> > > > > Vineeth
> > > > >
> > > > > On Thu, Dec 15, 2011 at 3:20 PM, Vineeth Mohan \<
> > > > > [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)\> wrote:
> > > > >
> > > > > > What is the general encoding of documents in internet , is it
> > > > > > ISO-8859-1 or something else.
> > > > > >
> > > > > > Thanks
> > > > > > Vineeth
> > > > > >
> > > > > > On Thu, Dec 15, 2011 at 1:58 PM, Karussell \<
> > > > > > [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)\> wrote:
> > > > > >
> > > > > > > > its replacing all such characters with question mark.
> > > > > > >
> > > > > > > That is a sign that your text is not UTF8. Make sure your source
> > > > > > > documents are what you guessed (latin1?) and that they get
> > > > > > > properly
> > > > > > > converted to UTF8. (I do not understand your converting code. Try
> > > > > > > to
> > > > > > > explain what you want :))
> > > > > > >
> > > > > > > Peter.
>
> --
> David Pilato
> [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
> Twitter : @dadoonet
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [December 16, 2011, 10:03am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/11 "2011-12-16T10:03:34Z")
Hi Vineeth
> My application when given a web page link , goes there get the text in
> there and gives that to elasticSearch.
>
> now lets say the link given is -
> [http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm\_source=feedburner&utm\_medium=feed&utm\_campaign=Feed%3A+fortunetermsheet+(Fortune+Finance%3A+Term+Sheet)](http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+fortunetermsheet+%28Fortune+Finance%3A+Term+Sheet%29)
>
> It fails while i sent it tru my application.
> Elasticsearch gives the following error -
> [gist:1485063 · GitHub](https://gist.github.com/1485063)
This problem has nothing to do with Elasticsearch. This is an issue
with your application not understanding encodings properly.
The page that you link to above claims to be UTF-8 (this may or may not
be true) but Firefox certainly seems to agree. That, of course, doesn't
mean that it doesn't have also include some illegal non-utf8 characters.
When you request the page, you get back a stream of bytes. The first
thing you need to do is to convert those bytes from whatever encoding
they are in to strings that your programming language can understand.
In Perl, for instance, we'd use the Encode module:
use Encode qw(decode);
my $bytes = get\_page($url);
my $content = decode('utf8',$bytes);
Now $content is a unicode string that your application understands,
rather than just a stream of bytes in one encoding or another.
Note: web pages quite often claim to be one encoding, but the content is
actually in a different encoding. In Perl, we would use something like
Encode::Guess to try to determine the real encoding
> **[Encode::Guess - Guesses encoding from data - metacpan.org](https://metacpan.org/pod/Encode::Guess)**
>
> Guesses encoding from data
Your mileage may vary 😉
clint
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 16, 2011, 10:28am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/12 "2011-12-16T10:28:30Z")
Thanks for you reply clint.
I tired the following methods
1. Guess the encoding using ICU and set that as the base encoding
2. use ICU Normalize
Both the above accomodated some of the non utf characters but not all.
Seems i need to invest lots more of my time on this.
I felt this was a common use case and someone would have found a silver
bullet to it.
Thanks
Vineeth
On Fri, Dec 16, 2011 at 3:33 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote:
> Hi Vineeth
>
> > My application when given a web page link , goes there get the text in
> > there and gives that to elasticSearch.
> >
> > now lets say the link given is -
>
> [http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm\_source=feedburner&utm\_medium=feed&utm\_campaign=Feed%3A+fortunetermsheet+(Fortune+Finance%3A+Term+Sheet)](http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+fortunetermsheet+%28Fortune+Finance%3A+Term+Sheet%29)
>
> > It fails while i sent it tru my application.
> > Elasticsearch gives the following error -
> > [gist:1485063 · GitHub](https://gist.github.com/1485063)
>
> This problem has nothing to do with Elasticsearch. This is an issue
> with your application not understanding encodings properly.
>
> The page that you link to above claims to be UTF-8 (this may or may not
> be true) but Firefox certainly seems to agree. That, of course, doesn't
> mean that it doesn't have also include some illegal non-utf8 characters.
>
> When you request the page, you get back a stream of bytes. The first
> thing you need to do is to convert those bytes from whatever encoding
> they are in to strings that your programming language can understand.
>
> In Perl, for instance, we'd use the Encode module:
>
> use Encode qw(decode);
> my $bytes = get\_page($url);
> my $content = decode('utf8',$bytes);
>
> Now $content is a unicode string that your application understands,
> rather than just a stream of bytes in one encoding or another.
>
> Note: web pages quite often claim to be one encoding, but the content is
> actually in a different encoding. In Perl, we would use something like
> Encode::Guess to try to determine the real encoding
> [https://metacpan.org/module/Encode::Guess](https://metacpan.org/module/Encode::Guess)
>
> Your mileage may vary 😉
>
> clint
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [December 16, 2011, 11:13am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/13 "2011-12-16T11:13:21Z")
Hi Vineeth
On Fri, 2011-12-16 at 15:58 +0530, Vineeth Mohan wrote:
> Thanks for you reply clint.
> I tired the following methods
> 1. Guess the encoding using ICU and set that as the base encoding
> 2. use ICU Normalize
> Both the above accomodated some of the non utf characters but not all.
> Seems i need to invest lots more of my time on this.
> I felt this was a common use case and someone would have found a
> silver bullet to it.
This is really something that you need to do in your application, before
you hit elasticsearch. ES expects to receive valid JSON (which must be
in UTF8), so your application needs to do the decoding and cleaning up
of the data. This is not something that can be handled in ES
clint
> Thanks
> Vineeth
>
> On Fri, Dec 16, 2011 at 3:33 PM, Clinton Gormley
> [clint@traveljury.com](mailto:clint@traveljury.com) wrote:
> Hi Vineeth
>
> ```
> > My application when given a web page link , goes there get
> the text in
> > there and gives that to elasticSearch.
> >
> > now lets say the link given is -
> >
> http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+fortunetermsheet+%28Fortune+Finance%3A+Term+Sheet%29
> >
> > It fails while i sent it tru my application.
> > ElasticSearch gives the following error -
> > https://gist.github.com/1485063
>
>
> This problem has nothing to do with ElasticSearch. This is an
> issue
> with your application not understanding encodings properly.
>
> The page that you link to above claims to be UTF-8 (this may
> or may not
> be true) but Firefox certainly seems to agree. That, of
> course, doesn't
> mean that it doesn't have also include some illegal non-utf8
> characters.
>
> When you request the page, you get back a stream of bytes.
> The first
> thing you need to do is to convert those bytes from whatever
> encoding
> they are in to strings that your programming language can
> understand.
>
> In Perl, for instance, we'd use the Encode module:
>
> use Encode qw(decode);
> my $bytes = get_page($url);
> my $content = decode('utf8',$bytes);
>
> Now $content is a unicode string that your application
> understands,
> rather than just a stream of bytes in one encoding or another.
>
> Note: web pages quite often claim to be one encoding, but the
> content is
> actually in a different encoding. In Perl, we would use
> something like
> Encode::Guess to try to determine the real encoding
> https://metacpan.org/module/Encode::Guess
>
> Your mileage may vary ;)
>
> clint
>
> ```
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 16, 2011, 2:31pm UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/14 "2011-12-16T14:31:57Z")
I tired everything i could google up or think with the little multithreaded
brain of mine.
All i could do is
change the error from
Caused by: org.elasticsearch.common.jackson.JsonParseException: Invalid
UTF-8 start byte 0xa0
to
Caused by: org.elasticsearch.common.jackson.JsonParseException: Invalid
UTF-8 middle byte 0x6e
Move the invalidiy of the byte from first to start to middle (cool naaa [?])
If anyone have the slightest idea where i am wrong , please point out.
The following code works better than all previous ones , but not on all
invalid UTF characters.
```
byte bytes[] = content.getBytes("UTF-8");
parsed = new String(bytes,"ISO-8859-1" );
```
Thanks
Vineeth
On Fri, Dec 16, 2011 at 4:43 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote:
> Hi Vineeth
>
> On Fri, 2011-12-16 at 15:58 +0530, Vineeth Mohan wrote:
>
> > Thanks for you reply clint.
> > I tired the following methods
> > 1. Guess the encoding using ICU and set that as the base encoding
> > 2. use ICU Normalize
> > Both the above accomodated some of the non utf characters but not all.
> > Seems i need to invest lots more of my time on this.
>
> > I felt this was a common use case and someone would have found a
> > silver bullet to it.
>
> This is really something that you need to do in your application, before
> you hit elasticsearch. ES expects to receive valid JSON (which must be
> in UTF8), so your application needs to do the decoding and cleaning up
> of the data. This is not something that can be handled in ES
>
> clint
>
> > Thanks
> > Vineeth
> >
> > On Fri, Dec 16, 2011 at 3:33 PM, Clinton Gormley
> > [clint@traveljury.com](mailto:clint@traveljury.com) wrote:
> > Hi Vineeth
> >
> > ```
> > > My application when given a web page link , goes there get
> > the text in
> > > there and gives that to elasticSearch.
> > >
> > > now lets say the link given is -
> > >
> >
> > ```
>
> [http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm\_source=feedburner&utm\_medium=feed&utm\_campaign=Feed%3A+fortunetermsheet+(Fortune+Finance%3A+Term+Sheet)](http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+fortunetermsheet+%28Fortune+Finance%3A+Term+Sheet%29)
>
> > ```
> > >
> > > It fails while i sent it tru my application.
> > > ElasticSearch gives the following error -
> > > https://gist.github.com/1485063
> >
> > This problem has nothing to do with ElasticSearch. This is an
> > issue
> > with your application not understanding encodings properly.
> >
> > The page that you link to above claims to be UTF-8 (this may
> > or may not
> > be true) but Firefox certainly seems to agree. That, of
> > course, doesn't
> > mean that it doesn't have also include some illegal non-utf8
> > characters.
> >
> > When you request the page, you get back a stream of bytes.
> > The first
> > thing you need to do is to convert those bytes from whatever
> > encoding
> > they are in to strings that your programming language can
> > understand.
> >
> > In Perl, for instance, we'd use the Encode module:
> >
> > use Encode qw(decode);
> > my $bytes = get_page($url);
> > my $content = decode('utf8',$bytes);
> >
> > Now $content is a unicode string that your application
> > understands,
> > rather than just a stream of bytes in one encoding or another.
> >
> > Note: web pages quite often claim to be one encoding, but the
> > content is
> > actually in a different encoding. In Perl, we would use
> > something like
> > Encode::Guess to try to determine the real encoding
> > https://metacpan.org/module/Encode::Guess
> >
> > Your mileage may vary ;)
> >
> > clint
> >
> > ```
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 17, 2011, 9:24am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/15 "2011-12-17T09:24:39Z")
Finally what i did
- Observed that all the special characters whcih were coming to me came
in this format in html source - Ř
- Before putting my data into ES , converted the special characters to
the above format.
- And all is well 🙂
Thanks
Vineeth
On Fri, Dec 16, 2011 at 8:01 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> I tired everything i could google up or think with the little
> multithreaded brain of mine.
> All i could do is
> change the error from
> Caused by: org.elasticsearch.common.jackson.JsonParseException: Invalid
> UTF-8 start byte 0xa0
> to
> Caused by: org.elasticsearch.common.jackson.JsonParseException: Invalid
> UTF-8 middle byte 0x6e
>
> Move the invalidiy of the byte from first to start to middle (cool naaa
> [?] )
>
> If anyone have the slightest idea where i am wrong , please point out.
>
> The following code works better than all previous ones , but not on all
> invalid UTF characters.
>
> ```
> byte bytes[] = content.getBytes("UTF-8");
> parsed = new String(bytes,"ISO-8859-1" );
>
> ```
>
> Thanks
> Vineeth
>
> On Fri, Dec 16, 2011 at 4:43 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote:
>
> > Hi Vineeth
> >
> > On Fri, 2011-12-16 at 15:58 +0530, Vineeth Mohan wrote:
> >
> > > Thanks for you reply clint.
> > > I tired the following methods
> > > 1. Guess the encoding using ICU and set that as the base encoding
> > > 2. use ICU Normalize
> > > Both the above accomodated some of the non utf characters but not all.
> > > Seems i need to invest lots more of my time on this.
> >
> > > I felt this was a common use case and someone would have found a
> > > silver bullet to it.
> >
> > This is really something that you need to do in your application, before
> > you hit elasticsearch. ES expects to receive valid JSON (which must be
> > in UTF8), so your application needs to do the decoding and cleaning up
> > of the data. This is not something that can be handled in ES
> >
> > clint
> >
> > > Thanks
> > > Vineeth
> > >
> > > On Fri, Dec 16, 2011 at 3:33 PM, Clinton Gormley
> > > [clint@traveljury.com](mailto:clint@traveljury.com) wrote:
> > > Hi Vineeth
> > >
> > > ```
> > > > My application when given a web page link , goes there get
> > > the text in
> > > > there and gives that to elasticSearch.
> > > >
> > > > now lets say the link given is -
> > > >
> > >
> > > ```
> >
> > [http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm\_source=feedburner&utm\_medium=feed&utm\_campaign=Feed%3A+fortunetermsheet+(Fortune+Finance%3A+Term+Sheet)](http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+fortunetermsheet+%28Fortune+Finance%3A+Term+Sheet%29)
> >
> > > ```
> > > >
> > > > It fails while i sent it tru my application.
> > > > ElasticSearch gives the following error -
> > > > https://gist.github.com/1485063
> > >
> > > This problem has nothing to do with ElasticSearch. This is an
> > > issue
> > > with your application not understanding encodings properly.
> > >
> > > The page that you link to above claims to be UTF-8 (this may
> > > or may not
> > > be true) but Firefox certainly seems to agree. That, of
> > > course, doesn't
> > > mean that it doesn't have also include some illegal non-utf8
> > > characters.
> > >
> > > When you request the page, you get back a stream of bytes.
> > > The first
> > > thing you need to do is to convert those bytes from whatever
> > > encoding
> > > they are in to strings that your programming language can
> > > understand.
> > >
> > > In Perl, for instance, we'd use the Encode module:
> > >
> > > use Encode qw(decode);
> > > my $bytes = get_page($url);
> > > my $content = decode('utf8',$bytes);
> > >
> > > Now $content is a unicode string that your application
> > > understands,
> > > rather than just a stream of bytes in one encoding or another.
> > >
> > > Note: web pages quite often claim to be one encoding, but the
> > > content is
> > > actually in a different encoding. In Perl, we would use
> > > something like
> > > Encode::Guess to try to determine the real encoding
> > > https://metacpan.org/module/Encode::Guess
> > >
> > > Your mileage may vary ;)
> > >
> > > clint
> > >
> > > ```
---
### Author:  [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [December 17, 2011, 2:45pm UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/16 "2011-12-17T14:45:07Z")
OK , so what was the real issue -
When i changed
StringEntity se = new StringEntity(params);
to
StringEntity se = new StringEntity(params,"UTF-8");
Everything went well.
Seems i had to specify charset while senting json to elasticSearch using
httpclient.
Thanks
Vineeth
On Sat, Dec 17, 2011 at 2:54 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
> Finally what i did
>
> - Observed that all the special characters whcih were coming to me
> came in this format in html source - Ř
> - Before putting my data into ES , converted the special characters to
> the above format.
> - And all is well 🙂
>
> Thanks
> Vineeth
>
> On Fri, Dec 16, 2011 at 8:01 PM, Vineeth Mohan [vineethmohan@algotree.com](mailto:vineethmohan@algotree.com)wrote:
>
> > I tired everything i could google up or think with the little
> > multithreaded brain of mine.
> > All i could do is
> > change the error from
> > Caused by: org.elasticsearch.common.jackson.JsonParseException: Invalid
> > UTF-8 start byte 0xa0
> > to
> > Caused by: org.elasticsearch.common.jackson.JsonParseException: Invalid
> > UTF-8 middle byte 0x6e
> >
> > Move the invalidiy of the byte from first to start to middle (cool naaa
> > [?] )
> >
> > If anyone have the slightest idea where i am wrong , please point out.
> >
> > The following code works better than all previous ones , but not on all
> > invalid UTF characters.
> >
> > ```
> > byte bytes[] = content.getBytes("UTF-8");
> > parsed = new String(bytes,"ISO-8859-1" );
> >
> > ```
> >
> > Thanks
> > Vineeth
> >
> > On Fri, Dec 16, 2011 at 4:43 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote:
> >
> > > Hi Vineeth
> > >
> > > On Fri, 2011-12-16 at 15:58 +0530, Vineeth Mohan wrote:
> > >
> > > > Thanks for you reply clint.
> > > > I tired the following methods
> > > > 1. Guess the encoding using ICU and set that as the base encoding
> > > > 2. use ICU Normalize
> > > > Both the above accomodated some of the non utf characters but not all.
> > > > Seems i need to invest lots more of my time on this.
> > >
> > > > I felt this was a common use case and someone would have found a
> > > > silver bullet to it.
> > >
> > > This is really something that you need to do in your application, before
> > > you hit elasticsearch. ES expects to receive valid JSON (which must be
> > > in UTF8), so your application needs to do the decoding and cleaning up
> > > of the data. This is not something that can be handled in ES
> > >
> > > clint
> > >
> > > > Thanks
> > > > Vineeth
> > > >
> > > > On Fri, Dec 16, 2011 at 3:33 PM, Clinton Gormley
> > > > [clint@traveljury.com](mailto:clint@traveljury.com) wrote:
> > > > Hi Vineeth
> > > >
> > > > ```
> > > > > My application when given a web page link , goes there get
> > > > the text in
> > > > > there and gives that to elasticSearch.
> > > > >
> > > > > now lets say the link given is -
> > > > >
> > > >
> > > > ```
> > >
> > > [http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm\_source=feedburner&utm\_medium=feed&utm\_campaign=Feed%3A+fortunetermsheet+(Fortune+Finance%3A+Term+Sheet)](http://finance.fortune.cnn.com/2011/12/14/investor-roundtable-experts/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+fortunetermsheet+%28Fortune+Finance%3A+Term+Sheet%29)
> > >
> > > > ```
> > > > >
> > > > > It fails while i sent it tru my application.
> > > > > ElasticSearch gives the following error -
> > > > > https://gist.github.com/1485063
> > > >
> > > > This problem has nothing to do with ElasticSearch. This is an
> > > > issue
> > > > with your application not understanding encodings properly.
> > > >
> > > > The page that you link to above claims to be UTF-8 (this may
> > > > or may not
> > > > be true) but Firefox certainly seems to agree. That, of
> > > > course, doesn't
> > > > mean that it doesn't have also include some illegal non-utf8
> > > > characters.
> > > >
> > > > When you request the page, you get back a stream of bytes.
> > > > The first
> > > > thing you need to do is to convert those bytes from whatever
> > > > encoding
> > > > they are in to strings that your programming language can
> > > > understand.
> > > >
> > > > In Perl, for instance, we'd use the Encode module:
> > > >
> > > > use Encode qw(decode);
> > > > my $bytes = get_page($url);
> > > > my $content = decode('utf8',$bytes);
> > > >
> > > > Now $content is a unicode string that your application
> > > > understands,
> > > > rather than just a stream of bytes in one encoding or another.
> > > >
> > > > Note: web pages quite often claim to be one encoding, but the
> > > > content is
> > > > actually in a different encoding. In Perl, we would use
> > > > something like
> > > > Encode::Guess to try to determine the real encoding
> > > > https://metacpan.org/module/Encode::Guess
> > > >
> > > > Your mileage may vary ;)
> > > >
> > > > clint
> > > >
> > > > ```
---
### Author:  [@anand](https://discuss.elastic.co/u/anand)
#### Post date: [March 3, 2014, 1:54am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/17 "2014-03-03T01:54:25Z")
Thanks Vineet, you saved my day
---
### Author:  [@riva](https://discuss.elastic.co/u/riva)
#### Post date: [May 10, 2016, 1:00am UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/18 "2016-05-10T01:00:46Z")
I'm having similar issues, but the ? marks only show up in one instance of elastic search but not another. On my local elastic search there is no problem with international characters, but on the elastic search in a different environment it shows up as ? marks. The code is the same, what could be causing the difference?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:52pm UTC](https://discuss.elastic.co/t/what-to-do-with-non-utf8-characters/6163/19 "2017-07-05T22:52:54Z")