Hello all!
I am currently using Filebeat to send Netflow events into Elasticsearch. At certain time stamps I can observe 100,000 to 3 million events per second.
Several of these events have the same data fields such as Source IP, Source Port, Destination IP, Destination port, Exporter timestamp, Event start, Event end, Number of Packets and Size of Packets sometimes around 400+ documents with the same fields.
Is there a way in filebeat where I can summarize documents with the same details into one document?