# How to aggregate data on elastic sent by logstash

**URL:** <https://discuss.elastic.co/t/how-to-aggregate-data-on-elastic-sent-by-logstash/205140>\
**Category:** Elasticsearch\
**Created:** [October 24, 2019, 7:22pm UTC](https://discuss.elastic.co/t/how-to-aggregate-data-on-elastic-sent-by-logstash/205140 "2019-10-24T19:22:28Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [November 1, 2019, 8:16pm UTC](https://discuss.elastic.co/t/how-to-aggregate-data-on-elastic-sent-by-logstash/205140/2 "2019-11-01T20:16:32Z")

</div>

Hi,

you should be able to do this using transform (see [https://www.elastic.co/guide/en/elasticsearch/reference/7.4/put-transform.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/put-transform.html)). The workflow would look like this:

1. you index the data into elasticsearch as single docs, this is the "source" index
2. you create a transform that pulls data from the "source" and aggregates it according to your needs into a "dest" index.

In your case you would group by entity Id, firm id, etc. and define aggregations, e.g. lastUpdated would be a max aggregation. Combining the docs is more tricky but doable using a scripted metric aggregation:

```auto
"associatedHouseholds": {
        "scripted_metric": {
          "init_script": "state.docs = []",
          "map_script": "state.docs.add(new HashMap(params['_source']))",
          "combine_script": "return state.docs",
          "reduce_script": "def ret = []; for (s in states) {for (d in s) { ret.add(d);}}return ret"
        }
      }

```

In order to do this as data comes in you need to sync source and dest via a timestamp field, it seems like lastUpdated is also the ingest timestamp and can therefore be used for this.

Note that you can instead of writing to an index write to a pipeline, this enables you to do further operations on the aggregated data.

I hope this helps!

---

_[View the full topic](https://discuss.elastic.co/t/how-to-aggregate-data-on-elastic-sent-by-logstash/205140)._