# Issues with encoding **URL:** \ **Category:** Beats\ **Created:** [November 20, 2015, 2:25pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160 "2015-11-20T14:25:43Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![vinod8427](https://avatars.discourse-cdn.com/v4/letter/v/b2d939/32.png) [@vinod8427](https://discuss.elastic.co/u/vinod8427)\ **Post date:** [November 20, 2015, 2:25pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/1 "2015-11-20T14:25:43Z")
I came across a major issue( atleast in my case ). The issue is as explained below and I am assuming it has to do with some sort of encoding. The logs in my case which are being parsed contain special characters and such. See below the actual line from my log: > START,Gooseberry-SA,0x0001FFC100DB7646,3068140336,GMT+05:30-Calcutta,11/20/2015,11:56:51.0,0,0,0,VoIP,IP-TO-IP,DEFAULT,,,9977554432,,0,,0,,0,,RL\_BELUR\_EXT\_EMS,1,GOOSEBERRY-SA:BELUR\_EXT\_EMS,10.54.26.102,10.54.80.143,BELUR\_INT\_EMS,,10.54.24.101:38262/10.54.80.140:6000,,10.54.26.101:38134/10.54.80.143:6000,0,,,0x0080023A,,,,2,"[SIP,1-32415@10.54.80.140](mailto:SIP,1-32415@10.54.80.140),%22sipp%22;tag=32415SIPpTag001,%22sut%22;tag=gK0cfd65f5,0,,,,sip:9977554432@10.54.24.102:5060,,,,sip:sipp@10.54.80.140:5555,sip:9977554432@10.54.24.102:5060,,,,,,,0,0,,0,0,,,,,,,,1,0,0,0,,,,,,,,0,,,,,,,,,0,,,,,,,,,,,",12,12,0,5,,,0x0a,9977554432,1,1,,BELUR\_EXT\_EMS,"SIP,474775912\_58694532@10.54.26.102,%22sipp%22;tag=gK0c7d662e,;tag=4829SIPpTag0114382283,0,,,,sip:+19977554432@10.54.80.143:6781;user=phone,,,,sip:sipp@10.54.26.102:5060,sip:10.54.80.143:6781;transport=UDP,,,,,,,0,0,,0,0,,,,,,,,1,0,0,0,,,,,,,,0,,,,,,,,,0,,,,,,,,,,,",,110,,,1,1,,,2,0x1C4C8168,0,,,,,,0,,,,1,,,,,,,6,,,,"sipp",2,1,1,1,1,,0,,,1,7,1,,,10.54.24.102,10.54.80.140,60121,16,8,,,,,,,,,0,,,TANDEM,,,,,,,13,1,,,,,,,,,,,,,,,,0,,,,,,,,0,,,"2,1,0,3",0,,,,,,,,,,,,,,,,,,,,,,,,,, See below the output received on logstash/redis: > "{"@timestamp":"2015-11-20T11:56:53.525Z","beat":{"hostname":"Gooseberry","name":"Gooseberry"},"count":1,"fields":{"sbc\_version":"5.0.0"},"input\_type":"log","message":"0,,,1,1,,,2,0x1C4C8168,0,,,,,,0,,,,1,,,,,,,6,,,,\"sipp\",2,1,1,1,1,,0,,,1,7,1,,,10.54.24.102,10.54.80.140,60121,16,8,,,,,,,,,0,,,TANDEM,,,,,,,13,1,,,,,,,,,,,,,,,,0,,,,,,,,0,,,\"2,1,0,3\",0,,,,,,,,,,,,,,,,,,,,,,,,,,\n143,BELUR\_INT\_EMS,,10.54.24.101:38262/10.54.80.140:6000,,10.54.26.101:38134/10.54.80.143:6000,0,,,0x0080023A,,,,2,\"[SIP,1-32415@10.54.80.140](mailto:SIP,1-32415@10.54.80.140),%22sipp%22\u003csip:sipp@10.54.80.140:5555\u003e;tag=32415SIPpTag001,%22sut%22\u003csip:9977554432@10.54.24.102:5060\u003e;tag=gK0cfd65f5,0,,,,sip:9977554432@10.54.24.102:5060,,,,sip:sipp@10.54.80.140:5555,sip:9977554432@10.54.24.102:5060,,,,,,,0,0,,0,0,,,,,,,,1,0,0,0,,,,,,,,0,,,,,,,,,0,,,,,,,,,,,\",12,12,0,5,,,0x0a,9977554432,1,1,,BELUR\_EXT\_EMS,\"[SIP,474775912\_58694532@10.54.26.102](mailto:SIP,474775912_58694532@10.54.26.102),%22sipp%22\u003csip:sipp@10.54.26.102:5060\u003e;tag=gK0c7d662e,\u003csip:+19977554432@10.54.80.143:6781;user=phone\u003e;tag=4829SIPpTag0114382283,0,,,,sip:+19977554432@10.54.80.143:6781;user=phone,,,,sip:sipp@10.54.26.102:5060,sip:10.54.80.143:6781;transport=UDP,,,,,,,0,0,,0,0,,,,,,,,1,0,0,0,,,,,,,,0,,,,,,,,,0,,,,,,,,,,,\",,110,,,1,1,,,2,0x1C4C8168,0,,,,,,0,,,,1,,,,,,,6,,,,\"sipp\",2,1,1,1,1,,0,,,1,7,1,,,10.54.24.102,10.54.80.140,60121,16,8,,,,,,,,,0,,,TANDEM,,,,,,,13,1,,,,,,,,,,,,,,,,0,,,,,,,,0,,,\"2,1,0,3\",0,,,,,,,,,,,,,,,,,,,,,,,,,,","offset":1897501,"source":"/var/log/sonus/sbx/evlog/1004A6C.ACT","type":"log"}" As seen above in the output message it has skipped some portion of the input from beginning and another concerning part is that the input message is not in the actual sequence. It has stitched the message string in some random fashion. Also the output message is in encoded form.I tried encoding set to plain, utf-8, utf-16be-bom etc but in vain. All I want is that the input message parsed to be returned as it is in the output. How could I do that? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 20, 2015, 2:30pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/2 "2015-11-20T14:30:05Z")
So would you describe as lines are partially truncated? We've identified an issue in rc2 with [very long log lines being incomplete](https://github.com/elastic/filebeat/issues/258). Updating to most recent nightlies should solve it. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 20, 2015, 2:42pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/3 "2015-11-20T14:42:45Z")
Link for nightlies: [https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/](https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/) ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 20, 2015, 2:48pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/4 "2015-11-20T14:48:23Z")
@vinod8427 Can you post your config here? What is the encoding of your files? ---
**Author:** ![vinod8427](https://avatars.discourse-cdn.com/v4/letter/v/b2d939/32.png) [@vinod8427](https://discuss.elastic.co/u/vinod8427)\ **Post date:** [November 20, 2015, 2:53pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/5 "2015-11-20T14:53:14Z")
The config used is as below: filebeat: prospectors: path: - /var/log/sonus/sbx/evlog/\*.ACT encoding: plain # Also tried utf-8 & utf-16be-bom ignore\_older: 20s scan\_frequency: 5s harvester\_buffer\_size: 2000000 output logstash: hosts: ["localhost:5044"] ---
**Author:** ![vinod8427](https://avatars.discourse-cdn.com/v4/letter/v/b2d939/32.png) [@vinod8427](https://discuss.elastic.co/u/vinod8427)\ **Post date:** [November 20, 2015, 3:05pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/6 "2015-11-20T15:05:32Z")
I have taken the input message and have tried to explain how the output message has been formed from it (See comments section in the image). ![](https://us1.discourse-cdn.com/elastic/original/2X/7/7e7c87ad45323dac43784205102b98b1bd855cd3.png) ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 20, 2015, 3:10pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/7 "2015-11-20T15:10:08Z")
so utf-16be-bom is not supported anymore. You can either try utf-16be (default utf16 over network) or utf-16le (default utf16 on windows) . Having the original log file it would be somewhat easier. Which operating system are you running on? Who is producing this log file (csv SIP CDRs?) ? Which locale is configured? digging into the lines I found this: original: ``` SIP,1-32415@10.54.80.140,%22sipp%22;tag=32415SIPpTag001,%22sut%22;tag=gK0cfd65f5,0,,,,sip:9977554432@10.54.24.102:5060,,,,sip:sipp@10.54.80.140:5555 ``` now 'truncated' one: ``` SIP,1-32415@10.54.80.140,%22sipp%22\u003csip:sipp@10.54.80.140 ``` Interestingly the ';' in first line is replaced by '\u003c' in unicode. The replacement is kinda correct, as '\u003c' stands for '\<'. But no idea why string contains '\<' character. From this point everything get's lost until the address: [sip:sipp@10.54.80.140](mailto:sip:sipp@10.54.80.140). If you use plain, the bytes are read as is. It is the json encoder escaping bad byte sequences. The decoder (configured by encoding) is responsible for reading files and translating all symbols into valid utf-8 for the json encoder not escaping wrong sequences. ---
**Author:** ![vinod8427](https://avatars.discourse-cdn.com/v4/letter/v/b2d939/32.png) [@vinod8427](https://discuss.elastic.co/u/vinod8427)\ **Post date:** [November 20, 2015, 3:20pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/8 "2015-11-20T15:20:05Z")
I am using Linux. The log files are SIP CDR's and generated by running some scripts. Locale used is IST. BTW I tried using utf-16be and its the same result I notice. Accordingly I guess I must be using plain encoding but as you mentioned the json encoder is messing things up. Logstash shipper used to handle this fine but I would like to migrate to filebeat instead. Please let me know if you require any further information from my end. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 20, 2015, 3:27pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/9 "2015-11-20T15:27:14Z")
@vinod8427 Can you send me one of these log files? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 20, 2015, 3:40pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/10 "2015-11-20T15:40:37Z")
So I created some tests locally about reading and writing said line and tests with multiple lines of random length, but can not reproduce the issue. Also json encoder seems to be fine with the line. Just press 'Run' on the sample encoding your line to json: [http://play.golang.org/p/Jee4wQst5t](http://play.golang.org/p/Jee4wQst5t) Maybe it's some interplay with other lines in your log file? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 20, 2015, 3:56pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/11 "2015-11-20T15:56:39Z")
can you get us the output of ``` $ locale ``` ? ---
**Author:** ![vinod8427](https://avatars.discourse-cdn.com/v4/letter/v/b2d939/32.png) [@vinod8427](https://discuss.elastic.co/u/vinod8427)\ **Post date:** [November 23, 2015, 6:12am UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/12 "2015-11-23T06:12:27Z")
locale output is as below: LANG=en\_US LANGUAGE= LC\_CTYPE="en\_US" LC\_NUMERIC="en\_US" LC\_TIME="en\_US" LC\_COLLATE="en\_US" LC\_MONETARY="en\_US" LC\_MESSAGES="en\_US" LC\_PAPER="en\_US" LC\_NAME="en\_US" LC\_ADDRESS="en\_US" LC\_TELEPHONE="en\_US" LC\_MEASUREMENT="en\_US" LC\_IDENTIFICATION="en\_US" LC\_ALL= ---
**Author:** ![vinod8427](https://avatars.discourse-cdn.com/v4/letter/v/b2d939/32.png) [@vinod8427](https://discuss.elastic.co/u/vinod8427)\ **Post date:** [November 23, 2015, 6:17am UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/13 "2015-11-23T06:17:57Z")
Shortly will be sending a mail with the attached log file. Sorry for the delay. ---
**Author:** ![vinod8427](https://avatars.discourse-cdn.com/v4/letter/v/b2d939/32.png) [@vinod8427](https://discuss.elastic.co/u/vinod8427)\ **Post date:** [November 23, 2015, 6:25am UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/14 "2015-11-23T06:25:06Z")
Sent a mail to Nicolas with the log attached. Steffen, could you message me your mail id? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 23, 2015, 7:20am UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/15 "2015-11-23T07:20:22Z")
@vinod8427 Thanks, got the e-mail. I forwarded it to Steffen. We will have a look and get back to you. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 23, 2015, 10:31am UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/16 "2015-11-23T10:31:48Z")
@vinod8427 I'm currently trying to reproduce this with the log file. I'm using FB -\> LS -\> ES. What is your setup? You list Redis above? Can you post your LS config? ---
**Author:** ![vinod8427](https://avatars.discourse-cdn.com/v4/letter/v/b2d939/32.png) [@vinod8427](https://discuss.elastic.co/u/vinod8427)\ **Post date:** [November 23, 2015, 10:36am UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/17 "2015-11-23T10:36:27Z")
I tried with both redis and LS and both received the same output. My LS config is pretty simple for the time being which is as below: > input { > #redis { host =\>"localhost" port =\>"6379" data\_type =\> "list" key =\>"filebeat" threads =\> 2 } > beats { > port =\> 5044 > } > } > filter { > grok { match =\> { "message" =\> "^START|^STOP|^ATTEMPT|^INTERMEDIATE|^REBOOT" } } > } > output { > stdout { codec=\>rubydebug } > } BTW my current config is FB-\>LS. If it goes well it would be FB-\>LS-\>ES ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 23, 2015, 1:36pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/18 "2015-11-23T13:36:40Z")
Can you also post your exact operating system? I think I have to test it on the same OS. Does this happen for all events (means as soon as you start filebeat) or does it only happen after a certain time? ---
**Author:** ![vinod8427](https://avatars.discourse-cdn.com/v4/letter/v/b2d939/32.png) [@vinod8427](https://discuss.elastic.co/u/vinod8427)\ **Post date:** [November 23, 2015, 1:41pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/19 "2015-11-23T13:41:19Z")
Its Debian OS 7.8. It happens everytime. Some events has slightly different outputs and never it is correct but the problem statement still holds the same for most of it. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 23, 2015, 1:43pm UTC](https://discuss.elastic.co/t/issues-with-encoding/35160/20 "2015-11-23T13:43:29Z")
Forgot to add to the list above: Exact LS and ES version. [Next page](https://discuss.elastic.co/t/issues-with-encoding/35160.md?page=2)