Json_encode returns null

Elastic Stack Logstash
1

I am parsing data from the pm2 manager for node and it works fine, however some of my data ends in a text field instead of an structure. This is my logstash configuration:

if [fields][type] == "pm2" {
        json {
            source => "message"
            target => "pm2"
        }
        json_encode {
            source => "pm2.message"
            target => "pm2.dataString"
        }
        json {
            source => "pm2.dataString"
            target => "pm2.data"
        }

        date {
            match => [ "timestamp", "yyyy-MM-dd HH:mm:ssZ" ]
            tag_on_failure => ["_dateparsefailure"]
        }
    }

Part of my template:

"pm2" : {
            "properties" : {
              "data" : {
                "type" : "object"
              },
              "dataString" : {
                "type" : "text"
              },
              "message" : {
                "type" : "text"
              },

Then some example data:

Original message: {"message":"{\n "className": "AppointmentBot",\n "dialog": "query",\n "step": 1,\n "millis": 0,\n "extra": "",\n "type": "logger",\n "level": "info",\n "message": "Query",\n "timestamp": "09-01-2018 15:13:05 +00:00"\n}\n","timestamp":"2018-01-09T15:13:05.310Z","type":"out","process_id":9,"app_name":"Chatbot-App-Dev"}

pm2.message ends up being this. which I don't want, I want every field separate: {
"className": "AppointmentBot",
"dialog": "query",
"step": 1,
"millis": 0,
"extra": "",
"type": "logger",
"level": "info",
"message": "Query",
"timestamp": "09-01-2018 15:13:05 +00:00"
}

So that is why I am trying to encode the JSON back to string and parse it to the data field which I made an object, but I am getting this that dataString is null.

Any idea? Thanks

2

I got this to work...
Note: you simply need to apply the json plugin to the message field twice, however if you have to continue with your approach then you should know that nested fields are referenced in this [a][b][c] style.

input {
  generator {
    message => '{"message":"{\n \"className\": \"AppointmentBot\",\n \"dialog\": \"query\",\n \"step\": 1,\n \"millis\": 0,\n \"extra\": \"\",\n \"type\": \"logger\",\n \"level\": \"info\",\n \"message\": \"Query\",\n \"timestamp\": \"09-01-2018 15:13:05 +00:00\"\n }\n ","timestamp":"2018-01-09T15:13:05.310Z","type":"out","process_id":9,"app_name":"Chatbot-App-Dev"}'
    count => 1
  }
}

filter {
  json {
    source => "message"
  }
  if "_jsonparsefailure" not in [tags]  {
    json {
      source => "message"
    }
  }
  if ! [Status] {
    mutate {
      update => { "Status" => "Failed" }
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

results

{
        "millis" => 0,
    "@timestamp" => 2018-01-09T16:20:07.127Z,
      "@version" => "1",
     "className" => "AppointmentBot",
      "app_name" => "Chatbot-App-Dev",
         "extra" => "",
          "host" => "Elastics-MacBook-Pro.local",
          "type" => "logger",
        "dialog" => "query",
          "step" => 1,
         "level" => "info",
     "timestamp" => "09-01-2018 15:13:05 +00:00",
      "sequence" => 0,
    "process_id" => 9,
       "message" => "Query"
}
2 Likes
3

Excellent! I owe you a beer!

My problem was indeed the [a][b] as the filters were not actually running. I ended up doing this:

 json {
    source => "message"
    target => "[pm2]"
    remove_field => "message"
  }
  if "_jsonparsefailure" not in [tags]  {
    json {
      source => "[pm2][message]"
      target => "[pm2][data]"
      remove_field => "[pm2][message]"
    }
  }

Now, is there a way to use the skip_on_invalid_json and still get the filter to tag the entry if it is invalid? My issue is that sometimes the message property is not a json and while everything works, I don't want to fill the logstash log with warnings.

Thanks again!

4

Just set skip_on_invalid_json to true in the json filter config.

json {
  source => "x"
  skip_on_invalid_json => true
}
1 Like
5

I would have sweared I tried that and it wasn't tagging the entries but I guessed I also played with the fields order and configuration. Everything working now. tips fedora

6

Yay, glad to help.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.