Transforming/filtering json logs

[swisscheese]

I have some WAF logs (from AWS) that are already in json format. I want to modify them with logstash in a few specific ways, and it’s not clear to me from the docs what the best way to do this is. Hoping someone can point me in the right direction.

Here’s an example entry:

{
"action": "ALLOW",
"formatVersion": 1,
"httpRequest": {
"args": "",
"clientIp": "1.2.3.4",
"country": "US",
"headers": [
{
"name": "Host",
"value": "www.example.com"
},
{
"name": "Content-Length",
"value": "890"
},
{
"name": "accept",
"value": "application/json"
},
{
"name": "content-type",
"value": "application/json"
},
{
"name": "cookie",
"value": "session=abcdefgh”
},
{
"name": "accept-encoding",
"value": "gzip,deflate"
},
{
"name": "user-agent",
"value": "curl 7.54.0"
}
],
"httpMethod": "POST",
"httpVersion": "HTTP/1.1",
"requestId": null,
"uri": “/upload”
}
"httpSourceName": "ALB",
"nonTerminatingMatchingRules": [
],
"rateBasedRuleList": [
],
"ruleGroupList": [
{
"excludedRules": null,
"nonTerminatingMatchingRules": [
],
"ruleGroupId": "",
"terminatingRule": null
}
],
"terminatingRuleId": "Default_Action",
"terminatingRuleMatchDetails": [
],
"terminatingRuleType": "REGULAR",
"timestamp": 1581541188137,
"webaclId": ""
}

What I would like logstash to send to ES in this case is this with the following changes:

1. Only keep what’s under httpRequest, throw away everything else and make that the top level object. Although I’ll want the timestamp field as well, and use that as @timestamp.

2. Reformat the way the headers are organized from name/values in an array from (for example) headers[0].name=“Host”, headers[0].value=“www.example.com”

to the more logical and searchable format of headers.Host=www.example.com, so something like

“headers”: {
“host”: “www.example.com”
...
}

3. And I probably want to get this confirming to Elastic Common Schema, but I can probably figure that out once I know what to use to accomplish 1 and 2.

Any pointers appreciated!