First, I want to thank @cleesmith for the wonderful unifiedbeat plugin, it works out of the box and it is a very robust application. I have only started it once I have installed it, and it is working for several weeks without any intervention.
unifiedbeat is used to read alerts from Unified2 binary files generated by snort and indexes them in Elasticsearch. These alerts are all indexed in the same document, at least daily, and has the following form:
( I am omitting the exact notation, just trying to show how it is stored)
event:1 (this is an 'event' typed record with id 1; like 'OpenSSL SSLv3 large heartbeat response')
packet:1:1 (this is a 'packet' typed record with id 1; it contains the network packet details for event:1)
event:2 (this is another 'event' typed record with id 2)
packet:2:2 (this is a 'packet' typed record with id 2; it contains the network packet details for event:2)
packet:2:3 (this is an another 'packet' typed record with id 3 belonging to the same event:2)
packet:2:4 (this is an another 'packet' typed record with id 4 belonging to the same event:2)
event:3 (this is another 'event' typed record with id 3)
packet:3:5 (this is a 'packet' typed record with id 5; it contains the network packet details for event:3)
etc...
So what I want to achieve is to get only the events from this index with their corresponding packet counts, in other words
where "key" is the "event_id" and "doc_count" is the count of packets for that "event_id".
And "event_id" is the only useful field that both record types have in common.
Also, if you have a lot of unified2 records in your indices you may want to limit the query
to something other than "match_all" ... maybe some time range.
Performed using Elasticsearch 2.2.0, and I'm sure there is a cleaner way to do this in Kibana but
I don't use it that often.
This returns the events in hits and packet counts off all events in aggregations sections, separately. Since all records are indexed in a daily bases, first I need to group in days, and then events in these days, global helps me to get all the aggregations regardless of the query results. Unfortunately there is no direct way to access the packet count of an event, but I first loop in aggs and create a ternary map:
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.