Parse rabbitmq json log

[ansamHox]

Hi, got json log message from rabbit as

{"timestamp":"2022-12-21 03:14:59.977922+02:00","level":"error","msg":"Error on AMQP connection <0.32551.1583>: enotconn (socket is not connected)","domain":"rabbitmq.connection","pid":"<0.32551.1583>"

JSONs are not parsed by default, i only got message field in ES, but I would like to extract 3 fields (actually 4)

*time as a timestamp in ISO8601 format, not this default RFC 3339 format
*loglevel as a text or keyword
*pid as a keyword without <> braces
*msg as a "message"

It should look in Kibana like this:

"time": "2022-12-21T02:14:59+01:00"
"level": "error"
"message": "Error on AMQP connection <0.32551.1583>: enotconn (socket is not connected)"
"pid": "0.32551.1583"

Tried to play with grok and json filters without luck converting timestamp from RFC 3339 to ISO8601, any input would be appreciated. Thanks.

[leandrojmp (Leandro Pereira)]

What does your Logstash configuration file looks like?

[leandrojmp (Leandro Pereira)]

If your json log looks like this:

{"timestamp":"2022-12-21 03:14:59.977922+02:00","level":"error","msg":"Error on AMQP connection <0.32551.1583>: enotconn (socket is not connected)","domain":"rabbitmq.connection","pid":"<0.32551.1583>"}

Then this filter will parse it:

filter {
    json {
        source => "message"
        remove_field => ["message"]
    }
    mutate {
        gsub => ["pid","<>",""]
        rename => {
            "msg" => "message"
        }
    }
    date {
        match => ["timestamp","yyyy-MM-dd HH:mm:ss.SSSSSSZZ"]
        target => "time"
        remove_field => ["timestamp"]
    }
}

The gsub will remove the <> from your pid field, and the rename will rename msg into message.

The date filter will parse your date format into ISO8601.

The output would be something like this:

{
    "@timestamp" => 2023-09-27T20:38:17.726537839Z,
      "@version" => "1",
           "pid" => "<0.32551.1583>",
        "domain" => "rabbitmq.connection",
          "time" => 2022-12-21T01:14:59.977Z,
         "level" => "error",
          "host" => "lab",
       "message" => "Error on AMQP connection <0.32551.1583>: enotconn (socket is not connected)"
}

[ansamHox]

why is it pid after gsub still with the brackets (<0.32551.1583>)

[leandrojmp (Leandro Pereira)]

Oh yeah, my mistake, I thought it worked on the first try.

Just need to change the gsub to this:

gsub => ["pid","[<>]",""]

And the <> will be removed.

[ansamHox]

thanks