# Aggregate records netflow

**URL:** <https://discuss.elastic.co/t/aggregate-records-netflow/139293>\
**Category:** Elasticsearch\
**Created:** [July 10, 2018, 8:23am UTC](https://discuss.elastic.co/t/aggregate-records-netflow/139293 "2018-07-10T08:23:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![eddie4](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@eddie4](https://discuss.elastic.co/u/eddie4)\
**Post date:** [July 10, 2018, 8:23am UTC](https://discuss.elastic.co/t/aggregate-records-netflow/139293/1 "2018-07-10T08:23:05Z")

</div>

**The problem:**  
We operate a elastic search cluster for storing netflow data. One of our clients operates a UDP service that is queried an insane amount by different IP's. This one client is responsible for 2/3 of the flows in elastic search. Adding about 3000 flows every second of every day.

**The question**  
Is there a way to aggregate all these records drop the source and destination IP and only calculate the amount of traffic used over the last 5secs? While deleting or better never storing the original records. There by reducing storage and processing time.

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [July 12, 2018, 9:29am UTC](https://discuss.elastic.co/t/aggregate-records-netflow/139293/2 "2018-07-12T09:29:28Z")

</div>

I haven't finished evaluation of it yet, but I am hopeful that the rollup API, introduced in 6.3, will facilitate this. If it works out I will be rolling the necessary setup into [ElastiFlow](https://github.com/robcowart/elastiflow).

---

<div class="post-metadata">

**Author:** ![eddie4](https://avatars.discourse-cdn.com/v4/letter/e/838e76/32.png) [@eddie4](https://discuss.elastic.co/u/eddie4)\
**Post date:** [July 14, 2018, 3:49pm UTC](https://discuss.elastic.co/t/aggregate-records-netflow/139293/3 "2018-07-14T15:49:24Z")

</div>

hmmm interesting we have completely custom built our platform. But I will look into ElastiFlow especially this feature.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 11, 2018, 3:49pm UTC](https://discuss.elastic.co/t/aggregate-records-netflow/139293/4 "2018-08-11T15:49:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.