Splitting JSON extracted fields for new-line behaves incorrectly vs JSON original message

I've an example dataset as below

{
  "org": "COMPANY11",
  "department": {
    "name": "Human Resources",
    "id": "HR"
  },
  "http_request": "Host: www.something.co.uk\r\nConnection: keep-alive\r\nUser-Agent: Mozilla/5.0 (Linux; Android 8.1.0; DUB-LX1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36\r\nAccept: image/webp,image/apng,image/*,*/*;q=0.8\r\nSec-Fetch-Site: same-origin\r\nSec-Fetch-Mode: no-cors\r\nSec-Fetch-Dest: image\r\nReferer: https://www.something.co.uk/assets/styles.css?20201008.2\r\nAccept-Encoding: gzip, deflate, br\r\nAccept-Language: en-GB,en-US;q=0.9,en;q=0.8\r\n",
  "src_ip": "217.182.97.145"
}

if I use a split on message it works. But if I extract message using JSON filter and then do on the extracted field the \r\n doesn't behave as it should be. I'm trying to split the http_request into key-value pairs based on : and \r\n Please find my pipeline config


input {
    file {
        path => "/tmp/json_sample.json"
        start_position => beginning
        sincedb_path => "/dev/null"
        exclude => "*.gz"
        codec => multiline {
            pattern => "^{$"
            negate => "true"
            what => "previous"
        }
    }
}

filter {
  json{
    source => "message"
  }
# did not work
 mutate {
    split => { "http_request" => "\\r\\n"}
 }
# did not work
  mutate {
    gsub => [ "http_request", "[\\\\]r", "rrr" ]
    gsub => [ "http_request", "[\\\\]n", "nnn" ]
  }

# did not work
    kv {
      field_split_pattern => "\\r\\n"
      value_split_pattern => ": "
      source => "http_request"
    }
}

output {
    stdout {
        codec => rubydebug
    }
}

If the http_request field contains the literal strings \r\n then

    mutate { gsub => [ "http_request", "\r", "" ] }
    mutate {
        split => { "http_request" => "
" }

To use kv you can

json { source => "message" remove_field => [ "message" ] }
mutate { gsub => [ "http_request", "\r", "" ] }
kv { field_split_pattern => "\n" value_split_pattern => ": " source => "http_request" }

would work.

didn't work Badger.
somehow after doing the json filter, the \r\n is not what it is supposed to be. can't take it out with any of the regex I've tried above

but before/pre json filter, I can do the gsub fine.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.