I want to aggregate some logs with different number, type and level of fields that have one common id field. Each line of log is relatively big and has complicated nested json structure. I've provided a simplified example of what they looks like, below:
2019-12-27 14:28:34,087 {"id":"123", "F1":{"A":0,"B":"abc"}, "F2":{"C":"def","F21":"{"D":"1","E":"2"}"}}
2019-12-27 14:28:34:132 {"id":"123", "F3":"xyz", "F4":{"G":0,"H":"abc"}, "F2":{"C":"def","F21":"{"D":"2","E":"2"}"}}
Duplicated key-values should be removed in the aggregated event and only one must be preserved, different values with the same key must also be grouped under the common key or being copied as-is but with the incremental key suffix, I mean key1:value1, key2:value2, ... . I must do all these without hard-coding the name of the fields, because they are changing frequently.
The aggregated event of the above logs must be like this:
2019-12-27 14:28:34,087 {"id":"123", "F1":{"A":0,"B":"abc"}, "F2":{"C":"def","F21":"{"D":{"1","2"},"E":"3"}"}, "F3":"xyz", "F4":{"G":0,"H":"abc"}}
I could to parse input logs using grok and json filters but I couldn't find any method to implement the rules that I've explained using aggregate filter. I guess implementing these rules needs iteration on the fields but as far as I know there is no loop concept in logsatsh. I did also do a small try with the ruby filter but no success!
How can I aggregate them? Is it possible at all?
p.s. Although there is no explicit criteria for determining beginning and end of a group but it is possible to group them with the timeout and inactivity_timeout.