Http://stackoverflow.com/questions/41427129/add-field-logstash-after-converting-value-to-json

Elastic Stack Logstash
1

1 - unable to fetch values inside JSON.
2 - unable to map timestamp from data.

For all general issues, please provide the following details for fast resolution:

  • Version:5.1.1
  • Operating System: Mac Sierra
  • Config File (if you have sensitive info, please remove it):

sample data

[debug] <0.545.0> <<"{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}">>

    input {
#    beats {
#        port => "5043"
#    }
     stdin {
        }
}
    filter{
             grok {
                match => { "message" => "%{TIMESTAMP_ISO8601:syslogtimestamp} [%{SYSLOG5424SD:logtype}] <% .      {BASE16FLOAT:processid}.0> <<%{QS:mydata}>>"}

        }
        json {
           source => "mydata"
           target => "parsedJson"
         add_field => {
      "user_id" => "%{[parsedJson][user_id]}}"
      "operation" => "%{[parsedJson][operation]}}"
      "data" => "%{[parsedJson][data]}}"
     }
#          remove_field=>["mydata"]
        }
       mutate {
          add_field => {
               "user_id" => "%{[parsedJson][user_id]}"
               "operation" => "%{[parsedJson][operation]}"
               "data" => "%{[parsedJson:data]}"
          }
        }
}
output {
        stdout {}
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

  • Sample Data:2016-12-28 00:16:40:53.290 [debug] <0.545.0> <<"{"user_id":"79","timestamp":"2016-12-28T11:10:26Z","operation":"ver3 - Requested for recommended,verified handle information","data":"","content_id":"","channel_id":""}">>

  • Steps to Reproduce:

second is timestamp not matching.

2

Where does the mydata come from and what does it look like? Make sure you format configuration and log samples as preformatted text.

Please also show a resulting event, preferably from a stdout { codec => rubydebug } output.

3

sample data is

[debug] <0.545.0> <<"{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}">>


{
   "@timestamp" => 2017-01-04T08:22:51.147Z,
         "data" => "%{[parsedJson][data]}}",
    "processid" => "0.545",
      "user_id" => "%{[parsedJson][user_id]}}",
     "@version" => "1",
         "host" => "Yatenders-iMac.local",
   "parsedJson" => "{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}",
      "message" => "[debug] <0.545.0> <<\"{\\\"user_id\\\":\\\"79\\\",\\\"timestamp\\\":\\\"2016-12-28T11:10:26Z\\\",\\\"operation\\\":\\\"ver3 - Requested for recommended,verified handle information\\\",\\\"data\\\":\\\"\\\",\\\"content_id\\\":\\\"\\\",\\\"channel_id\\\":\\\"\\\"}\">>",
    "operation" => "%{[parsedJson][operation]}}",
       "mydata" => "\"{\\\"user_id\\\":\\\"79\\\",\\\"timestamp\\\":\\\"2016-12-28T11:10:26Z\\\",\\\"operation\\\":\\\"ver3 - Requested for recommended,verified handle information\\\",\\\"data\\\":\\\"\\\",\\\"content_id\\\":\\\"\\\",\\\"channel_id\\\":\\\"\\\"}\"",
         "tags" => []
}

This is rubydebug

4

sample data

[debug] <0.545.0> <<"{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}">>

when my configuration is

input {
#    beats {
#        port => "5043"
#    }
     stdin {
        }
}
filter{
        grok {
           match => { "message" => "[%{SYSLOG5424SD:logtype}] <%{BASE16FLOAT:processid}.0> <<%{QS:mydata}>>"}

        }
#       date{
#       match => {"syslogtimestamp","ISO8601","yyyy-MM-dd HH:mm:ss:SSSZ"}
#       }
        json {
           source => "mydata"
           target => "parsedJson"
         add_field => {
      "user_id_up" => "%{[parsedJson][user_id]}}"
      "operation_up" => "%{[parsedJson][operation]}}"
      "data_up" => "%{[parsedJson][data]}}"
     }
#          remove_field=>["mydata"]
        }
        mutate {
           add_field => {
                "user_id" => "%{[parsedJson][user_id]}"
                "operation" => "%{[parsedJson][operation]}"
                "data" => "%{[parsedJson][data]}"
           }
         }
}
output {
        stdout {
                codec => rubydebug
        }
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

I am getting

{
            "data" => "%{[parsedJson][data]}",
         "message" => "[debug] <0.545.0> <<\"{\\\"user_id\\\":\\\"79\\\",\\\"timestamp\\\":\\\"2016-12-28T11:10:26Z\\\",\\\"operation\\\":\\\"ver3 - Requested for recommended,verified handle information\\\",\\\"data\\\":\\\"\\\",\\\"content_id\\\":\\\"\\\",\\\"channel_id\\\":\\\"\\\"}\">>",
          "mydata" => "\"{\\\"user_id\\\":\\\"79\\\",\\\"timestamp\\\":\\\"2016-12-28T11:10:26Z\\\",\\\"operation\\\":\\\"ver3 - Requested for recommended,verified handle information\\\",\\\"data\\\":\\\"\\\",\\\"content_id\\\":\\\"\\\",\\\"channel_id\\\":\\\"\\\"}\"",
            "tags" => [],
      "@timestamp" => 2017-01-04T08:36:35.631Z,
       "processid" => "0.545",
      "user_id_up" => "%{[parsedJson][user_id]}}",
         "data_up" => "%{[parsedJson][data]}}",
         "user_id" => "%{[parsedJson][user_id]}",
        "@version" => "1",
            "host" => "Yatenders-iMac.local",
      "parsedJson" => "{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}",
    "operation_up" => "%{[parsedJson][operation]}}",
       "operation" => "%{[parsedJson][operation]}"
}
5

Okay. mydata doesn't contain a JSON object but a double-quoted string that contains a JSON object. However, as it happens a double-quoted string can also be decoded as JSON so the result of your first json filter is a string with a JSON object (as you can see in your parsedJson field). If you send that string to another json filter you'll be okay.

6

Thanks a lot working fine now.

now I know use of rubydebug.

Thanks again

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.