Input file codec json UTF-16 parse error

Elastic Stack Logstash
1

Hi i am korean. We used logstash on a windows server.
logs are written in utf-16.
but Logstash does not correctly interpret json except the first line.

this is config file

input {
  file {
    path  =>
    start_position => 'beginning'
    codec => json {
      charset => 'UTF-16'
    }
  }
}
output {
  file {
    path =>
    codec => json_lines {
      charset => 'UTF-8'
    }
  }
}

and error message

[2017-01-06T08:55:59,899][ERROR][logstash.codecs.json ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('L' (code 76)): was expecting comma to separate OBJECT entries
at [Source: {"TimeStamp":"2017-01-06 15:57:51","Action":"LOGOUT","Uid":698,"NickName":"¸í��ì�´ì�¤ì�¢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�; line: 1, column: 84]>, :data=>"{"TimeStamp":"2017-01-06 15:57:51","Action":"LOGOUT","Uid":698,"NickName":"¸퉀왴잤숢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�"}
{"Action":"LOGIN","Description":"test","NickName":"트와이스","Cash":0,"TimeStamp":"2017-01-06 15:56:52","tags":,"Uid":698,"Coin":0,"path":"/home/ec2-user/USER/UTF16.json","ServerID":0,"@timestamp":"2017-01-06T08:55:59.882Z","@version":"1","host":"ip-172-31-14-82","Level":1,"Exp":0}
[2017-01-06T08:55:59,989][ERROR][logstash.codecs.json ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('L' (code 76)): was expecting comma to separate OBJECT entries
at [Source: {"TimeStamp":"2017-01-06 16:02:02","Action":"LOGIN","Uid":607,"NickName":"hí��ê¹�í��ì��ê°¢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�; line: 1, column: 84]>, :data=>"{"TimeStamp":"2017-01-06 16:02:02","Action":"LOGIN","Uid":607,"NickName":"h험깜헜숄갢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�"}
[2017-01-06T08:55:59,992][ERROR][logstash.codecs.json ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('L' (code 76)): was expecting comma to separate OBJECT entries
at [Source: {"TimeStamp":"2017-01-06 16:02:20","Action":"LOGIN","Uid":801,"NickName":"Ã�ì��í��ë��í�¢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�; line: 1, column: 83]>, :data=>"{"TimeStamp":"2017-01-06 16:02:20","Action":"LOGIN","Uid":801,"NickName":"Ì옄햙럈퀢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�"}
[2017-01-06T08:55:59,995][ERROR][logstash.codecs.json ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('L' (code 76)): was expecting comma to separate OBJECT entries
at [Source: {"TimeStamp":"2017-01-06 16:04:06","Action":"LOGIN","Uid":595,"NickName":"¨ë�©ì��í�¢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�; line: 1, column: 82]>, :data=>"{"TimeStamp":"2017-01-06 16:04:06","Action":"LOGIN","Uid":595,"NickName":"¨납옄혢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�"}
{"path":"/home/ec2-user/USER/UTF16.json","@timestamp":"2017-01-06T08:55:59.920Z","@version":"1","host":"ip-172-31-14-82","message":"{"TimeStamp":"2017-01-06 15:57:51","Action":"LOGOUT","Uid":698,"NickName":"¸퉀왴잤숢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�","tags":["_jsonparsefailure"]}

and output file

{"Action":"LOGOUT","Description":"test","NickName":"트와이스","Cash":0,"TimeStamp":"2017-01-06 15:57:51","tags":,"Uid":698,"Coin":0,"path":"D:/USER_20170106_155652_0000.json","ServerID":0,"@timestamp":"2017-01-06T09:00:46.491Z","@version":"1","host":"WP-SUNGJUN","Level":1,"Exp":0}
{"path":"D:/USER_20170106_155652_0000.json","@timestamp":"2017-01-06T09:00:49.594Z","@version":"1","host":"WP-SUNGJUN","message":"{"TimeStamp":"2017-01-06 16:02:02","Action":"LOGIN","Uid":607,"NickName":"h험깜헜숄갢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�","tags":["_jsonparsefailure"]}
{"path":"D:/USER_20170106_155652_0000.json","@timestamp":"2017-01-06T09:00:49.602Z","@version":"1","host":"WP-SUNGJUN","message":"{"TimeStamp":"2017-01-06 16:02:20","Action":"LOGIN","Uid":801,"NickName":"Ì옄햙럈퀢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�","tags":["_jsonparsefailure"]}
{"Action":"LOGIN","Description":"test","NickName":"soulaby","Cash":0,"TimeStamp":"2017-01-06 16:04:03","tags":,"Uid":250,"Coin":0,"path":"D:/USER_20170106_155652_0000.json","ServerID":0,"@timestamp":"2017-01-06T09:00:49.658Z","@version":"1","host":"WP-SUNGJUN","Level":1,"Exp":0}
{"path":"D:/USER_20170106_155652_0000.json","@timestamp":"2017-01-06T09:00:49.662Z","@version":"1","host":"WP-SUNGJUN","message":"{"TimeStamp":"2017-01-06 16:04:06","Action":"LOGIN","Uid":595,"NickName":"¨납옄혢,"Level":1,"Exp":0,"Coin":0,"Cash":0,"ServerID":0,"Description":"test"}�","tags":["_jsonparsefailure"]}

i want solution.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.