Help Parsing JSONFMT Rsyslog entries

[james_mtl]

I am having difficulty parsing the below.

Actual rsyslog line.

2018-06-18T12:36:55.588494-05:00 lb-workers.awsprod.com haproxy[2639]: 172.0.0.1:27947 [18/Jun/2018:12:36:55.587] application~ payment_backend/<NOSRV> 0/-1/-1/-1/0 5031593 - - SCNN 27/27/0/0/0 0/0 {50.0.0.0} "GET /v2/ping HTTP/1.1"

What's received by Logstash.

{  
  "message":"<13>Jun 18 12:36:55 10.0.0.0 {\"@timestamp\": \"2018-06-18T12:36:55.588494-05:00\",\"message\":\" 172.0.0.1:27947 [18\\/Jun\\/2018:12:36:55.587] application~ payment_backend\\/<NOSRV> 0\\/-1\\/-1\\/-1\\/0 503 1593 - - SCNN 27\\/27\\/0\\/0\\/0 0\\/0 {50.0.0.0} \\\"GET \\/v2\\/ping HTTP\\/1.1\\\"\",\"@fields\":{\"host\":\"lb-workers.awsprod.com\",\"syslog_facility\":\"local0\",\"syslog_facility_code\":\"16\",\"syslog_severity\":\"info\",\"syslog_severity_code\":\"6\",\"program\":\"haproxy\",\"pid\":\"2639\" }}"
}

Unescaped version of above.

{
  "message":"<13>Jun 18 12:36:55 10.0.0.0 {"@timestamp": "2018-06-18T12:36:55.588494-05:00","message":" 172.0.0.1:27947 [18/Jun/2018:12:36:55.587] application~ payment_backend/<NOSRV> 0/-1/-1/-1/0 503 1593 - - SCNN 27/27/0/0/0 0/0 {50.0.0.0} "GET /v2/ping HTTP/1.1"","@fields":{"host":"lb-workers.awsprod.com","syslog_facility":"local0","syslog_facility_code":"16","syslog_severity":"info","syslog_severity_code":"6","program":"haproxy","pid":"2639" }}"
}

What do I need to do and in what order?

• Parse "message" as JSON
• Unescape log line
• Apply GROK filter.

I believe the fields should correspond to a SYSLOG grok pattern but I"m having difficulty with the whole thing.

[james_mtl]

    if "syslog" in [tags] {
        grok {
          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:source_ip} %{GREEDYDATA:syslog_message}" }
        }
        
        json {
          source => "syslog_message"
        }
    }

Works but I'm having occasional parse errors. Message line occasionally contains an XML element and source IP field is not always present. Is there a more precise way to get the JSON object contained in the message line?

[magnusbaeck (Magnus Bäck)]

It looks like you have a syslog payload (beginning with "2018-06-18T12:36:55.588494-05:00") that some Logstash instance (?) is receiving and wrapping in JSON, then sending it via the syslog protocol to rsyslog, which wraps the whole thing in JSON. Can't you just dial back all the wrapping so you won't have to unwrap as much?

If that's not possible, use a json codec or json filter to unwrap the first level of JSON, then a grok filter to extract the second level of JSON from the syslog message, then run a json filter on that string.