Aggregate records netflow

Elastic Stack Elasticsearch
1

The problem:
We operate a elastic search cluster for storing netflow data. One of our clients operates a UDP service that is queried an insane amount by different IP's. This one client is responsible for 2/3 of the flows in elastic search. Adding about 3000 flows every second of every day.

The question
Is there a way to aggregate all these records drop the source and destination IP and only calculate the amount of traffic used over the last 5secs? While deleting or better never storing the original records. There by reducing storage and processing time.

2

I haven't finished evaluation of it yet, but I am hopeful that the rollup API, introduced in 6.3, will facilitate this. If it works out I will be rolling the necessary setup into ElastiFlow.

3

hmmm interesting we have completely custom built our platform. But I will look into ElastiFlow especially this feature.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.