[SOLVED] Split filter question a.k.a flatten json sub array


#1

Hi there!

I have a json input which looks like this:

[{
"Action": "COUNT",
"Timestamp": "2018-05-02T13:09:58Z",
"Request": {
"Country": "ES",
"URI": "/uploadMultiplePhotos.aspx",
"Headers": [{
> "Name": "Host",
> "Value": "www.example.net"
}, {
"Name": "Content-Length",
"Value": "226245"
}, {
"Name": "origin",
"Value": "https://www.example.net"
}, {
"Name": "user-agent",
"Value": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
}, {
"Name": "content-type",
"Value": "multipart/form-data; boundary=----WebKitFormBoundaryBnMGtRJgyfhSDZt3"
}, {
"Name": "accept",
"Value": "application/json"
}, {
"Name": "cache-control",
"Value": "no-cache"
}, {
"Name": "x-requested-with",
"Value": "XMLHttpRequest"
}, {
"Name": "x-ajax",
"Value": "example"
}, {
"Name": "referer",
"Value": "https://www.example.net/bla.aspx"
}, {
"Name": "accept-encoding",
"Value": "gzip, deflate, br"
}, {
"Name": "accept-language",
"Value": "es-ES,es;q=0.8"
}, {
"Name": "cookie",
"Value": "GREEDY"
}],
"ClientIP": "8.8.8.8",
"Method": "POST",
"HTTPVersion": "HTTP/2.0"
},
"Weight": 1
}]

and the logstash conf like this:

input {
file {
path => "/somepath/waf/*.log"
codec => "json"
discover_interval => 2
}
}
filter {
json {
source => "message"
}
date {
match => ["[Timestamp]", "ISO8601"]
target => "@timestamp"
remove_field => "timestamp"
}
split {
field => "[Request][Headers]"
}
}
output {
elasticsearch {
hosts => "https://someelk:443"
index => "waf-%{+YYYY.MM.dd}"
ssl_certificate_verification => "false"
}
}

Everything works except because Json array Name and Value are added as fields, but what i need is the "Name" key value added as fields and the "Value" key value xD as the Name field values

Ex of result wanted: a new field called Request.Header.Host and its value to be www.example.net

Any hints? Sorry that might be an easy one....

Thanks!


(Dave Martin) #2

Maybe a mutate? Add the value of Value to a filed with a name of Name? You run the risk of generating an arbitrary number of fields then.


#3

would you put an example?


#4

Field names are case sensitive, your field is named Timestamp, not timestamp.

mutate { add_field => { "%{[Request][Headers][Name]}" => "%{[Request][Headers][Value]}" } }

Will do it if I understood the ask correctly.


#5

sorry but no luck. its adding those fields but nothing else.
btw, thanks for the heads up in the timestamp typo


#6

Having re-read what you asked for, does this work?

mutate { add_field => { "[Request][Headers]%{[Request][Headers][Name]}" => "%{[Request][Headers][Value]}" } }

That gives you

       "Request" => {
           "ClientIP" => "8.8.8.8",
        "HTTPVersion" => "HTTP/2.0",
            "Country" => "ES",
                "URI" => "/uploadMultiplePhotos.aspx",
            "Headers" => {
             "Host" => "www.example.net",
             "Name" => "Host",
            "Value" => "www.example.net"
        },
             "Method" => "POST"
    },

Why do you split on [Request][Headers]?


#7

sorry to be a pain but still no luck.

here is how it looks the filter:

filter {
json {
source => "message"
}
date {
match => ["[Timestamp]", "ISO8601"]
target => "@timestamp"
remove_field => "Timestamp"
}
mutate {
add_field => {
"[Request][Headers]%{[Request][Headers][Name]}" => "%{[Request][Headers][Value]}"
}
}
}

i splitted request headers to test. but i didnt work


#8

So what does output { stdout { codec => rubydebug } } produce, and what don't you like about it?

BTW, if you have a json codec on the input, the json filter is not needed.


#9

thank you, i've removed the below now

json {
source => "message"
}

it makes no difference. it creates a single field

Request.Headers

with all the data inside :frowning:


#10

let me add that the json looks more like this, so it contains several objects

[{
"Action": "COUNT",
"Timestamp": "2018-05-02T15:36:14Z",
"Request": {
"Country": "PT",
"URI": "/uploadMultiplePhotos.aspx",
"Headers": [{
"Name": "Host",
"Value": "www.example.net"
}, {
"Name": "Content-Length",
"Value": "175253"
}, {
"Name": "origin",
"Value": "https://www.example.net"
}, {
"Name": "user-agent",
"Value": "Mozilla/5.0 (Linux; Android 7.0; LG-M200 Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36"
}, {
"Name": "content-type",
"Value": "multipart/form-data; boundary=----WebKitFormBoundaryXBuUMINHBoY9jHwN"
}, {
"Name": "accept",
"Value": "application/json"
}, {
"Name": "cache-control",
"Value": "no-cache"
}, {
"Name": "x-requested-with",
"Value": "XMLHttpRequest"
}, {
"Name": "save-data",
"Value": "on"
}, {
"Name": "x-ajax",
"Value": "blabla"
}, {
"Name": "referer",
"Value": "https://www.example.net/blabla"
}, {
"Name": "accept-encoding",
"Value": "gzip, deflate, br"
}, {
"Name": "accept-language",
"Value": "pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6"
}, {
"Name": "cookie",
"Value": "blabla"
}],
"ClientIP": "8.8.8.8",
"Method": "POST",
"HTTPVersion": "HTTP/2.0"
},
"Weight": 1
},{
"Action": "COUNT",
"Timestamp": "2018-05-02T15:36:14Z",
"Request": {
"Country": "PT",
"URI": "/uploadMultiplePhotos.aspx",
"Headers": [{
"Name": "Host",
"Value": "www.example.net"
}, {
"Name": "Content-Length",
"Value": "175253"
}, {
"Name": "origin",
"Value": "https://www.example.net"
}, {
"Name": "user-agent",
"Value": "Mozilla/5.0 (Linux; Android 7.0; LG-M200 Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36"
}, {
"Name": "content-type",
"Value": "multipart/form-data; boundary=----WebKitFormBoundaryXBuUMINHBoY9jHwN"
}, {
"Name": "accept",
"Value": "application/json"
}, {
"Name": "cache-control",
"Value": "no-cache"
}, {
"Name": "x-requested-with",
"Value": "XMLHttpRequest"
}, {
"Name": "save-data",
"Value": "on"
}, {
"Name": "x-ajax",
"Value": "blabla"
}, {
"Name": "referer",
"Value": "https://www.example.net/blabla"
}, {
"Name": "accept-encoding",
"Value": "gzip, deflate, br"
}, {
"Name": "accept-language",
"Value": "pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6"
}, {
"Name": "cookie",
"Value": "blabla"
}],
"ClientIP": "8.8.8.8",
"Method": "POST",
"HTTPVersion": "HTTP/2.0"
},
"Weight": 1
},{
"Action": "COUNT",
"Timestamp": "2018-05-02T15:36:14Z",
"Request": {
"Country": "PT",
"URI": "/uploadMultiplePhotos.aspx",
"Headers": [{
"Name": "Host",
"Value": "www.example.net"
}, {
"Name": "Content-Length",
"Value": "175253"
}, {
"Name": "origin",
"Value": "https://www.example.net"
}, {
"Name": "user-agent",
"Value": "Mozilla/5.0 (Linux; Android 7.0; LG-M200 Build/NRD90U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36"
}, {
"Name": "content-type",
"Value": "multipart/form-data; boundary=----WebKitFormBoundaryXBuUMINHBoY9jHwN"
}, {
"Name": "accept",
"Value": "application/json"
}, {
"Name": "cache-control",
"Value": "no-cache"
}, {
"Name": "x-requested-with",
"Value": "XMLHttpRequest"
}, {
"Name": "save-data",
"Value": "on"
}, {
"Name": "x-ajax",
"Value": "blabla"
}, {
"Name": "referer",
"Value": "https://www.example.net/blabla"
}, {
"Name": "accept-encoding",
"Value": "gzip, deflate, br"
}, {
"Name": "accept-language",
"Value": "pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6"
}, {
"Name": "cookie",
"Value": "blabla"
}],
"ClientIP": "8.8.8.8",
"Method": "POST",
"HTTPVersion": "HTTP/2.0"
},
"Weight": 1
}]


#11

mmmm just payed attention on the [ and ] at the beginning and at the end of the json... :thinking:


#12

So what does output { stdout { codec => rubydebug } } produce, and what don't you like about it?

My guess is you will end up wanting something like

  ruby {
    code => '
      event.get("[Request][Headers]").each { |a|
        name = a["Name"]
        value = a["Value"]
        event.set( "[Request][HeadersFlattened]#{name}", value)
      }
    '
  }

Which will get you this if your event is a single JSON object from that array.

        "HeadersFlattened" => {
                      "accept" => "application/json",
                "content-type" => "multipart/form-data; boundary=----WebKitFormBoundaryBnMGtRJgyfhSDZt3",
                      "cookie" => "GREEDY",
                      "origin" => "https://www.example.net",
                  "user-agent" => "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36",
               "cache-control" => "no-cache",
            "x-requested-with" => "XMLHttpRequest",
                        "Host" => "www.example.net",
                     "referer" => "https://www.example.net/bla.aspx",
              "Content-Length" => "226245",
             "accept-encoding" => "gzip, deflate, br",
             "accept-language" => "es-ES,es;q=0.8",
                      "x-ajax" => "example"
        },

Filtering json with embedded xml
Format object with key value
#13

yep that made the trick :crazy_face::smiley:. however just for the first of the objects.... then the rest look like this:

"Headers" => [
[ 0] {
"Name" => "Host",
"Value" => "www.example.net"
},
[ 1] {
"Name" => "Content-Length",
"Value" => "90726"
},
[ 2] {
"Name" => "origin",
"Value" => "https://www.example.net"
},
[ 3] {
"Name" => "user-agent",
"Value" => "Mozilla/5.0 (Linux; Android 7.0; RNE-L21 Build/HUAWEIRNE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36"
},
[ 4] {
"Name" => "content-type",
"Value" => "multipart/form-data; boundary=----WebKitFormBoundaryU0dmYmrvc8iBUZ8C"
},
[ 5] {
"Name" => "accept",
"Value" => "application/json"
},
[ 6] {
"Name" => "cache-control",
"Value" => "no-cache"
},
[ 7] {
"Name" => "x-requested-with",
"Value" => "XMLHttpRequest"
},
[ 8] {
"Name" => "x-ajax",
"Value" => "blabla"
},
[ 9] {
"Name" => "referer",
"Value" => "https://www.example.net/blae.aspx"
},
[10] {
"Name" => "accept-encoding",
"Value" => "gzip, deflate, br"
},
[11] {
"Name" => "accept-language",
"Value" => "es-ES,es;q=0.9"
},
[12] {
"Name" => "cookie",
"Value" => "blabla"
}
]


#14

oh wait i think because i need to remove "[Request][Headers]" after being processed by ruby block.... let me try!


#15

yes!!!! thank you badger you rock!


(system) #16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.