# How to aggregate a complicated json log?

**URL:** <https://discuss.elastic.co/t/how-to-aggregate-a-complicated-json-log/213249>\
**Category:** Logstash\
**Created:** [December 28, 2019, 7:14am UTC](https://discuss.elastic.co/t/how-to-aggregate-a-complicated-json-log/213249 "2019-12-28T07:14:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ilia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilia/32/60042_2.png) [@Ilia](https://discuss.elastic.co/u/Ilia)\
**Post date:** [December 28, 2019, 7:14am UTC](https://discuss.elastic.co/t/how-to-aggregate-a-complicated-json-log/213249/1 "2019-12-28T07:14:40Z")

</div>

I want to aggregate some logs with different number, type and level of fields that have one common `id` field. Each line of log is relatively big and has complicated nested json structure. I've provided a simplified example of what they looks like, below:

```
2019-12-27 14:28:34,087 {"id":"123", "F1":{"A":0,"B":"abc"}, "F2":{"C":"def","F21":"{"D":"1","E":"2"}"}}
2019-12-27 14:28:34:132 {"id":"123", "F3":"xyz", "F4":{"G":0,"H":"abc"}, "F2":{"C":"def","F21":"{"D":"2","E":"2"}"}}

```

Duplicated key-values should be removed in the aggregated event and only one must be preserved, different values with the same key must also be grouped under the common key or being copied as-is but with the incremental key suffix, I mean `key1:value1, key2:value2, ... ` . I must do all these without hard-coding the name of the fields, because they are changing frequently.

The aggregated event of the above logs must be like this:

```
2019-12-27 14:28:34,087 {"id":"123", "F1":{"A":0,"B":"abc"}, "F2":{"C":"def","F21":"{"D":{"1","2"},"E":"3"}"}, "F3":"xyz", "F4":{"G":0,"H":"abc"}}

```

I could to parse input logs using `grok` and `json` filters but I couldn't find any method to implement the rules that I've explained using `aggregate` filter. I guess implementing these rules needs iteration on the fields but as far as I know there is no `loop` concept in logsatsh. I did also do a small try with the `ruby` filter but no success!

How can I aggregate them? Is it possible at all?

p.s. Although there is no explicit criteria for determining beginning and end of a group but it is possible to group them with the `timeout` and `inactivity_timeout`.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 25, 2020, 7:14am UTC](https://discuss.elastic.co/t/how-to-aggregate-a-complicated-json-log/213249/2 "2020-01-25T07:14:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.