We encounter some difficulties in mapping the following JSON array
{
"docs": [
{
"_source": {
.... ,
"httpRequest": {
"clientIp": "REDACTED",
"country": "REDACTED",
"headers": [
{
"name": "host",
"value": "REDACTED"
},
{
"name": "accept-language",
"value": "REDACTED"
},
{
"name": "operid",
"value": "REDACTED"
},
{
"name": "user-agent",
"value": "REDACTED"
},
{
"name": "content-type",
"value": "REDACTED"
},
{
"name": "clientid",
"value": "REDACTED"
},
{
"name": "accept",
"value": "*/*"
},
{
"name": "sec-fetch-site",
"value": "REDACTED"
},
{
"name": "accept-encoding",
"value": "REDACTED"
},
{
"name": "cookie",
"value": "_gcl_au=REDACTED; utm_source=REDACTED; utm_term=REDACTED; device-id=REDACTED; sb_country=REDACTED; phone=REDACTED; _gcl_aw=REDACTED; locale=REDACTED; deviceId=REDACTED; usrId=REDACTED; userCert=350; accessToken=REDACTED; refreshToken=REDACTED; userId=REDACTED; _ga=REDACTED"
}
],
"uri": "REDACTED",
"args": "REDACTED",
"httpVersion": "HTTP/2.0",
"httpMethod": "GET",
"requestId": "REDACTED"
},
....
}
}
]
}
The raw log originates from AWS WAF, and we employ AWS WAF integration to transmit it to Elastic. We used managed pipeline for processing these logs in which was "logs-aws.waf-1.28.3". While we successfully extract the header, it's stored in JSON format:
My objective is to store each items within the "headers" array as it's own event in Elasticsearch, something similar to the following
{
"docs": [
{
"_source": {
.... ,
"httpRequest": {
"clientIp": "REDACTED",
"country": "REDACTED",
"headers.host": "REDACTED",
"headers.accept-language": "REDACTED",
"headers.operid": "REDACTED",
"headers.user-agent": "REDACTED",
...
"uri": "REDACTED",
"args": "REDACTED",
"httpVersion": "HTTP/2.0",
"httpMethod": "GET",
"requestId": "REDACTED"
},
....
}
}
]
}
Is it possible to do so using ingest pipeline ? Any help would be very appreciated