i'm having trouble wrapping my head around how i would go about transforming some items in a log line into nested objects (or whether it's even worthwhile)
its a custom NGINX log that logs upstream connections and the time to response, and there are times when i end up with multiple upstream destinations and response times in one log line, like this:
2019-12-10T12:23:23-05:00 | 10.0.0.1, 10.0.0.2 | [0.06, 0.07] | api.whatever.com
Where:
10.0.0.1 and response time 0.06 are associated and
10.0.0.2 and response time 0.07 are associated
so I was thinking an array of nested objects would be ideal here.
@timestamp: "2019-12-10T12:23:23-05:00", nginx.gateway.upstream: [ { ip: 10.0.0.1, response_time: 0.06 }, { ip: 10.0.0.2, response_time: 0.07 } ], host: "api.whatever.com"
when i parse the log, I can parse the IPs into a list variable: nginx.gateway.upstream.ip_list
and the response_times into a similar list: nginx.gateway.upstream.response_time_list
but i'm not sure how to go from those lists to nested objects. log entries may have only one upstream, or multiple.
i think its worth maintaining the associations so i can trace delays back to the appropriate upstream source.
i guess i should add that I'm using an ingest pipeline to do the parsing and transformations.