We operate a elastic search cluster for storing netflow data. One of our clients operates a UDP service that is queried an insane amount by different IP's. This one client is responsible for 2/3 of the flows in elastic search. Adding about 3000 flows every second of every day.
Is there a way to aggregate all these records drop the source and destination IP and only calculate the amount of traffic used over the last 5secs? While deleting or better never storing the original records. There by reducing storage and processing time.