Filter JSON content in field and parse target with existing mapping

Hi,
I'm trying to filter a second type of messages coming in my input queue.
The difference between messages is that in the second type of messages the log is embedded in a JSON as value of the "Message" field. I configured the JSON filter plugin as follow:

filter {
  json {
    source => "Message"
    target => "doc"
    remove_field => ["Message"]
  }
}

As expected the JSON parsed value of "Message" is placed in the new field "doc". Now, I'm struggling to further parse the "doc" nested fields. I'm using the add_field parameter to do this, without success tho.

Can you show us what the doc field looks like and explain exactly what you want to parse from it?

Here's an example:

{
      "filename" => "fixtures/sqs.py",
    "@timestamp" => 2018-05-17T15:34:00.215Z,
         "cg_id" => "c6b46234-74d1-45dd-ab3e-1402b2f7147b",
      "@version" => "1",
          "name" => "mex-test",
       "message" => "This is a manually inserted message",
           "env" => "testing",
     "levelname" => "INFO",
        "region" => "local",
      "customer" => "TestCustomerCG",
          "tags" => [
        [0] "_dateparsefailure"
    ]
}
{
          "Type" => "Notification",
    "@timestamp" => 2018-05-17T15:34:00.268Z,
      "@version" => "1",
          "json" => "{\"_id\": null, \"levelname\": \"INFO\", \"asctime\": \"2018/05/11 16:02:14.209597\", \"customer\": \"TestCustomerCG\", \"message\": \"This is a manually inserted message as wrapped by SNS\", \"name\": \"test-logger\", \"funcName\": \"<module>\", \"filename\": \"fixtures/sqs.py\", \"env\": \"testing\", \"region\": \"local\"}"

So basically I need the nested field in "json" to be unwrapped and shown as the fields of the first message.

Have you tried using "codec" => "json" in your output plugin?

Yes, nothing changes

So you have json in a field called json. Use a json filter to parse it.

json { source => "json" target => "someField" }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.