How to gsub \ in json message

Elastic Stack Logstash
1

I have a message below that is failing due to jsonparse error. If I remove all the \ (backslashes) from the message in text editor and check in json parser, it parsed it properly. Now, to get rid of the \ in the json message filed, I tried the mutate + gsub as below (from the official docs) with [\] option which is not working! Need some help with this:

config file:

filter {
    mutate {
      gsub => [ "message", "[\\]", "" ]  # << 2 backslashes
}
}

       [0] "_jsonparsefailure"
    ],
        "message" => [
        [ 0] "{\"node_id_str\":\"xxxx\",\"subscription_id_str\":\"Sub2\",\"encoding_path\":\"Cisco-IOS-XR-shellutil-oper:system-time/uptime\",\"collection_id\":\"151649\",\"collection_start_time\":\"1560440022382\",\"msg_timestamp\":\"1560440022396\",\"data_json\":[{\"timestamp\":\"1560440022394\",\"keys\":[],\"content\":{\"host-name\":\"xxxx\",\"uptime\":6117928}}],\"collection_end_time\":\"1560440022396\"}",

So, how do I get rid of the \ in this message? Interestingly, if I try to match it and replace with some char B, it does it next to the \ so not sure what this is matching....

filter {
   mutate {
     gsub => [ "message", "[\\]", "B" ] # << 2 back slashes
 }
}

      "message" => [
        [0] "{B\"node_id_strB\":B\"xxxx-IE1B\",B\"subscription_id_strB\":B\"Sub2B\",B\"encoding_pathB\":B\"Cisco-IOS-XR-shellutil-oper:system-time/uptimeB\",B\"collection_idB\":B\"151843B\",B\"collection_start_timeB\":B\"1560447583109B\",B\"msg_timestampB\":B\"1560447583123B\",B\"data_jsonB\":[{B\"timestampB\":B\"1560447583120B\",B\"keysB\":[],B\"contentB\":{B\"host-nameB\":B\"xxxx-IE1B\",B\"uptimeB\":6125489}}],B\"collection_end_timeB\":B\"1560447583123B\"}",
        [1] "{B\"node_id_strB\":B\"xxxx-IE1B\",B\"subscription_id_strB\":B\"Sub2B\",B\"encoding_pathB\":B\"Cisco-IOS-XR-shellutil-oper:system-time/uptimeB\",B\"collection_idB\":B\"151844B\",B\"collection_start_timeB\":B\"1560447613123B\",B\"msg_timestampB\":B\"1560447613136B\",B\"data_jsonB\":[{B\"timestampB\":B\"1560447613132B\",B\"keysB\":[],B\"contentB\":{B\"host-nameB\":B\"xxxx-IE1B\",B\"uptimeB\":6125519}}],B\"collection_end_timeB\":B\"1560447613136B\"}"

[EDIT: added code-fences around code blocks to increase readability -- @yaauie]

2

If that is from rubydebug your message does not contain backslash, it is just rubydebug escaping the double quotes. Not sure how your replacement with B could occur in that case.

3

:-). If that is from ruby debug output plugin, why does logstash fail to parse the json?

4

What does the json filter configuration look like?

5

filter {
mutate {
split => ["message", " " ]
}
}

filter {
mutate {
gsub => [ "message", '\"', '"' ]
}
}

6

Do you not have a json filter configured? logstash is not going to parse the JSON unless you tell it to.

7

I have tried codec json and json_lines hoping that it will parse the incoming lines as json. I even added json filter to the end of the config before output but no luck parsing the message to json format. This is what I see when I run logstash:

[WARN ] 2019-06-14 12:38:45.267 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>["{"node_id_str":"xxxx","subscription_id_str":"Sub2","encoding_path":"Cisco-IOS-XR-shellutil-oper:system-time/uptime","collection_id":"152429","collection_start_time":"1560515908973","msg_timestamp":"1560515908990","data_json":[{"timestamp":"1560515908987","keys":,"content":{"host-name":"xxxx","uptime":6193815}}],"collection_end_time":"1560515908990"}"], :exception=>java.lang.ClassCastException: org.jruby.RubyArray cannot be cast to org.jruby.RubyIO}
[WARN ] 2019-06-14 12:38:45.272 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>["{"node_id_str":"xxxx","subscription_id_str":"Sub2","encoding_path":"Cisco-IOS-XR-shellutil-oper:system-time/uptime","collection_id":"152429","collection_start_time":"1560515908973","msg_timestamp":"1560515908990","data_json":[{"timestamp":"1560515908987","keys":,"content":{"host-name":"xxxx","uptime":6193815}}],"collection_end_time":"1560515908990"}"], :exception=>java.lang.ClassCastException: org.jruby.RubyArray cannot be cast to org.jruby.RubyIO}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
"tags" => [
[0] "_jsonparsefailure"
],
"port" => 48700,
"@timestamp" => 2019-06-14T12:38:44.988Z,
"host" => "xxxx-v0.xxxx.com",
"@version" => "1",
"message" => [
[0] "{"node_id_str":"xxxx","subscription_id_str":"Sub2","encoding_path":"Cisco-IOS-XR-shellutil-oper:system-time/uptime","collection_id":"152429","collection_start_time":"1560515908973","msg_timestamp":"1560515908990","data_json":[{"timestamp":"1560515908987","keys":,"content":{"host-name":"xxxx","uptime":6193815}}],"collection_end_time":"1560515908990"}"
],
"type" => "jsonotcp"
}

8

How did you configure the json filter?

9

filter {
json {
source => "message"
target => "jsonmessage"
}
}

10

You used a mutate+split filter to convert message to an array, so that should be

json {
    source => "[message][0]"
    target => "jsonmessage"
}
11

That worked for the first element! How do I make it parse every element in the array?

{
"tags" => [
[0] "_jsonparsefailure"
],
"jsonmessage" => {
"collection_end_time" => "1560531678180",
"node_id_str" => "xxxx",
"collection_id" => "152579",
"encoding_path" => "Cisco-IOS-XR-shellutil-oper:system-time/uptime",
"subscription_id_str" => "Sub2",
"collection_start_time" => "1560531678163",
"msg_timestamp" => "1560531678180",
"data_json" => [
[0] {
"timestamp" => "1560531678176",
"content" => {
"host-name" => "xxxx",
"uptime" => 6209584
},
"keys" =>
}
]
},
"@timestamp" => 2019-06-14T17:01:25.510Z,
"@version" => "1",
"port" => 34850,
"message" => [
[0] "{"node_id_str":"xxxx","subscription_id_str":"Sub2","encoding_path":"Cisco-IOS-XR-shellutil-oper:system-time/uptime","collection_id":"152579","collection_start_time":"1560531678163","msg_timestamp":"1560531678180","data_json":[{"timestamp":"1560531678176","keys":,"content":{"host-name":"xxxx","uptime":6209584}}],"collection_end_time":"1560531678180"}"
],
"host" => "xxxx-v0.xxxx.com",
"type" => "jsonotcp"
}

12

I think you would have to use ruby for that.

13

Can you share the code to do so?

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.