Query to list Snort alerts indexed by unifiedbeat

Elastic Stack Elasticsearch
1

Hi all,

First, I want to thank @cleesmith for the wonderful unifiedbeat plugin, it works out of the box and it is a very robust application. I have only started it once I have installed it, and it is working for several weeks without any intervention.

unifiedbeat is used to read alerts from Unified2 binary files generated by snort and indexes them in Elasticsearch. These alerts are all indexed in the same document, at least daily, and has the following form:

( I am omitting the exact notation, just trying to show how it is stored)

event:1 (this is an 'event' typed record with id 1; like 'OpenSSL SSLv3 large heartbeat response')
packet:1:1 (this is a 'packet' typed record with id 1; it contains the network packet details for event:1)
event:2 (this is another 'event' typed record with id 2)
packet:2:2 (this is a 'packet' typed record with id 2; it contains the network packet details for event:2)
packet:2:3 (this is an another 'packet' typed record with id 3 belonging to the same event:2)
packet:2:4 (this is an another 'packet' typed record with id 4 belonging to the same event:2)
event:3 (this is another 'event' typed record with id 3)
packet:3:5 (this is a 'packet' typed record with id 5; it contains the network packet details for event:3)
etc...

So what I want to achieve is to get only the events from this index with their corresponding packet counts, in other words

event:1 : 1 packet
event:2 : 3 packets
event:3: 1 packet
etc...

I am new to ES, so excuse me if I use the wrong terms, feel free to ask for more details if needed.

Kind Regards

2

I'm glad to hear that unifiedbeat is helpful.
Since there are two record types within the same index, this is the only way I know of doing counts:

curl -XPOST "http://your.ip:9200/unifiedbeat-*/packet/_search" -d'
{
"size":0,
"query": {"match_all":{}},
"aggs": {
  "count_packets_per_event_id": { 
    "terms": {
      "field": "event_id"
    }
  }
}
}'

which returns:

. . . 
"aggregations": {
      "count_packets_per_event_id": {
         "doc_count_error_upper_bound": 2,
         "sum_other_doc_count": 190,
         "buckets": [
            {
               "key": 1,
               "doc_count": 7
            },
            {
               "key": 2,
               "doc_count": 7
            },
. . .

where "key" is the "event_id" and "doc_count" is the count of packets for that "event_id".
And "event_id" is the only useful field that both record types have in common.

Also, if you have a lot of unified2 records in your indices you may want to limit the query
to something other than "match_all" ... maybe some time range.
Performed using Elasticsearch 2.2.0, and I'm sure there is a cleaner way to do this in Kibana but
I don't use it that often.

Hope this helps.
Chris

3

Thank you Chris,

I have worked on this a little bit more, and come up with the following solution:

{  
   'query':{  
      'match':{  
         '_type':'event'
      }
   },
.....   
   'aggs':{  
      'all_records':{  
         'global':{  },
         'aggs':{  
            'days':{  
               'terms':{  
                  'field':'_index',
                  'size':'0'
               },
               'aggs':{  
                  'packets':{  
                     'terms':{  
                        'field':'event_id',
                        'size':'0'
                     }
                  }
               }
            }
         }
      }
   }
}

This returns the events in hits and packet counts off all events in aggregations sections, separately. Since all records are indexed in a daily bases, first I need to group in days, and then events in these days, global helps me to get all the aggregations regardless of the query results. Unfortunately there is no direct way to access the packet count of an event, but I first loop in aggs and create a ternary map:

packet_counts[_index][event_id] = number_of_packet

I couldn't come up with a cleaner solution, but I am still working on it.

Kind Regards,