Flatten complex json-array-structure (.har-file / http-archive)

Elastic Stack Logstash
1

I am trying to "flatten" a nested array-structure given in a .HAR-File (httparchive).

Incoming JSON (.har-File)
{
"log": {
"version": "1.2",
"creator": {
"name": "mitmproxy har_dump",
"version": "0.1",
"comment": "mitmproxy version mitmproxy 4.0.4"
},
"entries": [
{
"startedDateTime": "2018-08-29T15:39:39.122213+00:00",
"time": 736,
"request": {
"method": "GET",
"url": "https://www.heise.de/",
"httpVersion": "HTTP/1.1",
"cookies": [
{
"name": "wt3_eid",
"value": "%3B288689636920174%7C2153019600282217739%232153544712961480642",
"httpOnly": false,
"secure": false
},
{
"name": "wt_rla",
"value": "288689636920174%2C1%2C1535447129164",
"httpOnly": false,
"secure": false
}
],
"headers": [
{
"name": "Host",
"value": "www.heise.de"
},
{
"name": "Cache-Control",
"value": "max-age=0"
}
],
"queryString": [],
"headersSize": 901,
"bodySize": 0
},
"response": {
"status": 200,
"statusText": "OK",
"httpVersion": "HTTP/1.1",
"cookies": [],
"headers": [
{
"name": "Server",
"value": "nginx"
},
{
"name": "Date",
"value": "Wed, 29 Aug 2018 15:39:33 GMT"
}
],
"content": {
"size": 50383,
"compression": 279904,
"mimeType": "text/html; charset=UTF-8",
"text": ""
},
"redirectURL": "",
"headersSize": 506,
"bodySize": 50383
},
"cache": {},
"timings": {
"send": 5,
"receive": 48,
"wait": 28,
"connect": 235,
"ssl": 420
}
},
{ }
]
}
}

Ruby-Filter
filter {
ruby {
init => ""
code => "

if !("splitted_entries" in [tags]) {
    json {
        source => "message"
    }
    split { 
        field => "entries"
        add_tag => ["splitted_entries"]
    }
}

if ( "splitted_entries" in [tags] and [entries][request] ) {
    split {
        field => "entries[request]"
    }

    event.get('[request][cookies]').each { |cookie|
         event.set(cookie['name'], cookie['value'])
    }
 }

            "
     }

}

But this does not work at all! I am not sure if I am even close achieving a ".HAR-to-Indexable-Data-Transformation". Those gave me some doubts:

I appreciate any help even a "stop trying & stick to java or python for pre-procesing".

Thx,
Tobi

2

You can't have Logstash's configuration language inside the code block of your ruby filter. Only ruby code goes there.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.