I have huge amount of logs coming from Cisco ASA firewall specially firewall deny. for example i have more than 10 messages from same source IP accessing same destination with same destination port. So the idea is , instead sending 10 different messages to elastic, i want to send one aggregated message with the additional field of total_hits.
my original message is like below (multiple within short period of time, say 10 event per second)
This creates a new event containing whatever you have inserted into the map. Exactly what you are seeing.
You could have the aggregate filter append a field to every event, but it would only be the number of hits it has seen so far, not the total number of hits. You cannot expect the filter to look into the future.
You might be able to do that by using aggregate and then using the output of the aggregate to create upserts to each document that has previously been inserted into elasticsearch.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.