# Logstash Invalid Character for UTF-16/Unicode encoding **URL:** \ **Category:** Logstash\ **Created:** [July 29, 2016, 7:02am UTC](https://discuss.elastic.co/t/logstash-invalid-character-for-utf-16-unicode-encoding/56702 "2016-07-29T07:02:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![sreeram](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@sreeram](https://discuss.elastic.co/u/sreeram)\ **Post date:** [July 29, 2016, 7:02am UTC](https://discuss.elastic.co/t/logstash-invalid-character-for-utf-16-unicode-encoding/56702/1 "2016-07-29T07:02:20Z")
Hi, Below is the logstash Input plugin configured to read logs generated by my Windows Application, input { file { path =\> ["D:/ELK/LoggerTestApp/Server/Logger/GetAssetPointerById/\*\*/\*.txt"] codec =\> plain { charset =\> "UTF-16" } sincedb\_path =\> ["D:/ELK/logstash/since.db"] start\_position =\> "beginning" } } My Application is configured to write logs in Unicode/UTF-16 encoding format only. Once the logs are read & ported to ElasticSearch, I'm seeing a invalid character (�) in each log as shown below ![](https://us1.discourse-cdn.com/elastic/original/2X/0/0821c2bcd8a07e23045d5ce55f442e98855b77f7.png) Please advice me on how to avoid these invalid characters. **Environment Details** 1. _Operating System_ : Win 7, Win 2008 R2 2. _ElasticSearch_ : elasticsearch-2.3.4 3. _Logstash_ : logstash-2.3.4 4. _Kibana_ : kibana-4.5.3-windows Thanks in Advance. ---
**Author:** ![Matthew\_Isett](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matthew_isett/32/57770_2.png) [@Matthew\_Isett](https://discuss.elastic.co/u/Matthew_Isett)\ **Post date:** [October 26, 2016, 1:29pm UTC](https://discuss.elastic.co/t/logstash-invalid-character-for-utf-16-unicode-encoding/56702/2 "2016-10-26T13:29:08Z")
Did you resolve this? Did you try a different charset, like charset =\> "ISO-8859-1"? Or are you looking for elasticsearch to handle the charset differently: [https://www.elastic.co/guide/en/elasticsearch/guide/current/unicode-normalization.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/unicode-normalization.html) ---
**Author:** ![sreeram](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@sreeram](https://discuss.elastic.co/u/sreeram)\ **Post date:** [November 7, 2016, 7:35am UTC](https://discuss.elastic.co/t/logstash-invalid-character-for-utf-16-unicode-encoding/56702/3 "2016-11-07T07:35:27Z")
Hi Matthew, Thanks for your response. We're still living with the Invalid character issue. I'll try the suggestions provided . ---
**Author:** ![mkm](https://avatars.discourse-cdn.com/v4/letter/m/edb3f5/32.png) [@mkm](https://discuss.elastic.co/u/mkm)\ **Post date:** [May 28, 2017, 9:12am UTC](https://discuss.elastic.co/t/logstash-invalid-character-for-utf-16-unicode-encoding/56702/4 "2017-05-28T09:12:22Z")
Hi, did you resolve this? ---
**Author:** ![CDR](https://avatars.discourse-cdn.com/v4/letter/c/d9b06d/32.png) [@CDR](https://discuss.elastic.co/u/CDR)\ **Post date:** [June 20, 2017, 5:38pm UTC](https://discuss.elastic.co/t/logstash-invalid-character-for-utf-16-unicode-encoding/56702/5 "2017-06-20T17:38:05Z")
Was there any resolution here? I am currently dealing with the same situation. ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [June 21, 2017, 2:11pm UTC](https://discuss.elastic.co/t/logstash-invalid-character-for-utf-16-unicode-encoding/56702/6 "2017-06-21T14:11:21Z")
The question-mark-in-black-diamond character is a replacement character that is used when the UTF16 -\> UTF8 character conversion fails. This piece of config `codec => plain { charset => "UTF-16" }` says to Logstash "Treat all text as UTF16 and convert it to UTF8" There may be some illegal surrogates [http://unicode.org/faq/utf\_bom.html#utf16-7](http://unicode.org/faq/utf_bom.html#utf16-7) or maybe the charset conversion library we use does not deal with noncharacters [http://www.unicode.org/faq/private\_use.html#noncharacters](http://www.unicode.org/faq/private_use.html#noncharacters) very well. ---
**Author:** ![Kurt\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kurt_s/32/17755_2.png) [@Kurt\_S](https://discuss.elastic.co/u/Kurt_S)\ **Post date:** [January 11, 2018, 4:35pm UTC](https://discuss.elastic.co/t/logstash-invalid-character-for-utf-16-unicode-encoding/56702/7 "2018-01-11T16:35:35Z")
Hi, I had the same kind of problem: a "�" (black diamond or cube) at the end of each line after converting from UTF-16. I don't know what the character was (carriage return, vertical tab,... - something like that) but I did not need it. I worked around the problem by stripping the "�" from the message field: ``` mutate { gsub => ["message","�",""] } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 4:25am UTC](https://discuss.elastic.co/t/logstash-invalid-character-for-utf-16-unicode-encoding/56702/8 "2022-11-04T04:25:30Z")