[SOLVED] Split filter question a.k.a flatten json sub array

Elastic Stack Logstash
1

Hi there!

I have a json input which looks like this:

[{
"Action": "COUNT",
"Timestamp": "2018-05-02T13:09:58Z",
"Request": {
"Country": "ES",
"URI": "/uploadMultiplePhotos.aspx",
"Headers": [{
> "Name": "Host",
> "Value": "www.example.net"
}, {
"Name": "Content-Length",
"Value": "226245"
}, {
"Name": "origin",
"Value": "https://www.example.net"
}, {
"Name": "user-agent",
"Value": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
}, {
"Name": "content-type",
"Value": "multipart/form-data; boundary=----WebKitFormBoundaryBnMGtRJgyfhSDZt3"
}, {
"Name": "accept",
"Value": "application/json"
}, {
"Name": "cache-control",
"Value": "no-cache"
}, {
"Name": "x-requested-with",
"Value": "XMLHttpRequest"
}, {
"Name": "x-ajax",
"Value": "example"
}, {
"Name": "referer",
"Value": "https://www.example.net/bla.aspx"
}, {
"Name": "accept-encoding",
"Value": "gzip, deflate, br"
}, {
"Name": "accept-language",
"Value": "es-ES,es;q=0.8"
}, {
"Name": "cookie",
"Value": "GREEDY"
}],
"ClientIP": "8.8.8.8",
"Method": "POST",
"HTTPVersion": "HTTP/2.0"
},
"Weight": 1
}]

and the logstash conf like this:

input {
file {
path => "/somepath/waf/*.log"
codec => "json"
discover_interval => 2
}
}
filter {
json {
source => "message"
}
date {
match => ["[Timestamp]", "ISO8601"]
target => "@timestamp"
remove_field => "timestamp"
}
split {
field => "[Request][Headers]"
}
}
output {
elasticsearch {
hosts => "https://someelk:443"
index => "waf-%{+YYYY.MM.dd}"
ssl_certificate_verification => "false"
}
}

Everything works except because Json array Name and Value are added as fields, but what i need is the "Name" key value added as fields and the "Value" key value xD as the Name field values

Ex of result wanted: a new field called Request.Header.Host and its value to be www.example.net

Any hints? Sorry that might be an easy one....

Thanks!

2

Maybe a mutate? Add the value of Value to a filed with a name of Name? You run the risk of generating an arbitrary number of fields then.

3

would you put an example?

4

Field names are case sensitive, your field is named Timestamp, not timestamp.

mutate { add_field => { "%{[Request][Headers][Name]}" => "%{[Request][Headers][Value]}" } }

Will do it if I understood the ask correctly.

5

sorry but no luck. its adding those fields but nothing else.
btw, thanks for the heads up in the timestamp typo

6

Having re-read what you asked for, does this work?

mutate { add_field => { "[Request][Headers]%{[Request][Headers][Name]}" => "%{[Request][Headers][Value]}" } }

That gives you

       "Request" => {
           "ClientIP" => "8.8.8.8",
        "HTTPVersion" => "HTTP/2.0",
            "Country" => "ES",
                "URI" => "/uploadMultiplePhotos.aspx",
            "Headers" => {
             "Host" => "www.example.net",
             "Name" => "Host",
            "Value" => "www.example.net"
        },
             "Method" => "POST"
    },

Why do you split on [Request][Headers]?

7

sorry to be a pain but still no luck.

here is how it looks the filter:

filter {
json {
source => "message"
}
date {
match => ["[Timestamp]", "ISO8601"]
target => "@timestamp"
remove_field => "Timestamp"
}
mutate {
add_field => {
"[Request][Headers]%{[Request][Headers][Name]}" => "%{[Request][Headers][Value]}"
}
}
}

i splitted request headers to test. but i didnt work

8

So what does output { stdout { codec => rubydebug } } produce, and what don't you like about it?

BTW, if you have a json codec on the input, the json filter is not needed.

9

thank you, i've removed the below now

json {
source => "message"
}

it makes no difference. it creates a single field

Request.Headers

with all the data inside :frowning:

10

let me add that the json looks more like this, so it contains several objects

[{
"Action": "COUNT",
"Timestamp": "2018-05-02T15:36:14Z",
"Request": {
"Country": "PT",
"URI": "/uploadMultiplePhotos.aspx",
"Headers": [{
"Name": "Host",
"Value": "www.example.net"
}, {
"Name": "Content-Length",
"Value": "175253"
}, {
"Name": "origin",
"Value": "https://www.example.net"
}, {
"Name": "user-agent",
"Value": "Mozilla/5.0 (Linux; Android 7.0; LG-M200 Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36"
}, {
"Name": "content-type",
"Value": "multipart/form-data; boundary=----WebKitFormBoundaryXBuUMINHBoY9jHwN"
}, {
"Name": "accept",
"Value": "application/json"
}, {
"Name": "cache-control",
"Value": "no-cache"
}, {
"Name": "x-requested-with",
"Value": "XMLHttpRequest"
}, {
"Name": "save-data",
"Value": "on"
}, {
"Name": "x-ajax",
"Value": "blabla"
}, {
"Name": "referer",
"Value": "https://www.example.net/blabla"
}, {
"Name": "accept-encoding",
"Value": "gzip, deflate, br"
}, {
"Name": "accept-language",
"Value": "pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6"
}, {
"Name": "cookie",
"Value": "blabla"
}],
"ClientIP": "8.8.8.8",
"Method": "POST",
"HTTPVersion": "HTTP/2.0"
},
"Weight": 1
},{
"Action": "COUNT",
"Timestamp": "2018-05-02T15:36:14Z",
"Request": {
"Country": "PT",
"URI": "/uploadMultiplePhotos.aspx",
"Headers": [{
"Name": "Host",
"Value": "www.example.net"
}, {
"Name": "Content-Length",
"Value": "175253"
}, {
"Name": "origin",
"Value": "https://www.example.net"
}, {
"Name": "user-agent",
"Value": "Mozilla/5.0 (Linux; Android 7.0; LG-M200 Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36"
}, {
"Name": "content-type",
"Value": "multipart/form-data; boundary=----WebKitFormBoundaryXBuUMINHBoY9jHwN"
}, {
"Name": "accept",
"Value": "application/json"
}, {
"Name": "cache-control",
"Value": "no-cache"
}, {
"Name": "x-requested-with",
"Value": "XMLHttpRequest"
}, {
"Name": "save-data",
"Value": "on"
}, {
"Name": "x-ajax",
"Value": "blabla"
}, {
"Name": "referer",
"Value": "https://www.example.net/blabla"
}, {
"Name": "accept-encoding",
"Value": "gzip, deflate, br"
}, {
"Name": "accept-language",
"Value": "pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6"
}, {
"Name": "cookie",
"Value": "blabla"
}],
"ClientIP": "8.8.8.8",
"Method": "POST",
"HTTPVersion": "HTTP/2.0"
},
"Weight": 1
},{
"Action": "COUNT",
"Timestamp": "2018-05-02T15:36:14Z",
"Request": {
"Country": "PT",
"URI": "/uploadMultiplePhotos.aspx",
"Headers": [{
"Name": "Host",
"Value": "www.example.net"
}, {
"Name": "Content-Length",
"Value": "175253"
}, {
"Name": "origin",
"Value": "https://www.example.net"
}, {
"Name": "user-agent",
"Value": "Mozilla/5.0 (Linux; Android 7.0; LG-M200 Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36"
}, {
"Name": "content-type",
"Value": "multipart/form-data; boundary=----WebKitFormBoundaryXBuUMINHBoY9jHwN"
}, {
"Name": "accept",
"Value": "application/json"
}, {
"Name": "cache-control",
"Value": "no-cache"
}, {
"Name": "x-requested-with",
"Value": "XMLHttpRequest"
}, {
"Name": "save-data",
"Value": "on"
}, {
"Name": "x-ajax",
"Value": "blabla"
}, {
"Name": "referer",
"Value": "https://www.example.net/blabla"
}, {
"Name": "accept-encoding",
"Value": "gzip, deflate, br"
}, {
"Name": "accept-language",
"Value": "pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6"
}, {
"Name": "cookie",
"Value": "blabla"
}],
"ClientIP": "8.8.8.8",
"Method": "POST",
"HTTPVersion": "HTTP/2.0"
},
"Weight": 1
}]

11

mmmm just payed attention on the [ and ] at the beginning and at the end of the json... :thinking:

12

So what does output { stdout { codec => rubydebug } } produce, and what don't you like about it?

My guess is you will end up wanting something like

  ruby {
    code => '
      event.get("[Request][Headers]").each { |a|
        name = a["Name"]
        value = a["Value"]
        event.set( "[Request][HeadersFlattened]#{name}", value)
      }
    '
  }

Which will get you this if your event is a single JSON object from that array.

        "HeadersFlattened" => {
                      "accept" => "application/json",
                "content-type" => "multipart/form-data; boundary=----WebKitFormBoundaryBnMGtRJgyfhSDZt3",
                      "cookie" => "GREEDY",
                      "origin" => "https://www.example.net",
                  "user-agent" => "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36",
               "cache-control" => "no-cache",
            "x-requested-with" => "XMLHttpRequest",
                        "Host" => "www.example.net",
                     "referer" => "https://www.example.net/bla.aspx",
              "Content-Length" => "226245",
             "accept-encoding" => "gzip, deflate, br",
             "accept-language" => "es-ES,es;q=0.8",
                      "x-ajax" => "example"
        },
4 Likes
13

yep that made the trick :crazy_face::smiley:. however just for the first of the objects.... then the rest look like this:

"Headers" => [
[ 0] {
"Name" => "Host",
"Value" => "www.example.net"
},
[ 1] {
"Name" => "Content-Length",
"Value" => "90726"
},
[ 2] {
"Name" => "origin",
"Value" => "https://www.example.net"
},
[ 3] {
"Name" => "user-agent",
"Value" => "Mozilla/5.0 (Linux; Android 7.0; RNE-L21 Build/HUAWEIRNE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36"
},
[ 4] {
"Name" => "content-type",
"Value" => "multipart/form-data; boundary=----WebKitFormBoundaryU0dmYmrvc8iBUZ8C"
},
[ 5] {
"Name" => "accept",
"Value" => "application/json"
},
[ 6] {
"Name" => "cache-control",
"Value" => "no-cache"
},
[ 7] {
"Name" => "x-requested-with",
"Value" => "XMLHttpRequest"
},
[ 8] {
"Name" => "x-ajax",
"Value" => "blabla"
},
[ 9] {
"Name" => "referer",
"Value" => "https://www.example.net/blae.aspx"
},
[10] {
"Name" => "accept-encoding",
"Value" => "gzip, deflate, br"
},
[11] {
"Name" => "accept-language",
"Value" => "es-ES,es;q=0.9"
},
[12] {
"Name" => "cookie",
"Value" => "blabla"
}
]

14

oh wait i think because i need to remove "[Request][Headers]" after being processed by ruby block.... let me try!

15

yes!!!! thank you badger you rock!

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.