Help with aggregating events

[TehMuffinMoo]

Hey Guys,

My first time posting here, but I’m hoping somebody may be able to assist in pointing me in the right direction.

I’ve got OpenVPN logs going into my log server and there’s a number of events which get logged periodically. They are almost identical except for the last values which I currently assign to individual fields using the kv plugin, but this means I get 9 events each exactly the same except for one field.

An example of the batch of events is as follows;

Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_VER=2.4.4
Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_PLAT=win
Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_PROTO=2
Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_LZ4=1
Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_LZ4v2=1
Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_LZO=1
Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_COMP_STUB=1
Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_COMP_STUBv2=1
Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_TCPNL=1
Jan 17 13:49:53 openvpn[51830]: username/1.1.1.1:61595 peer info: IV_GUI_VER=OpenVPN_GUI_11

A snippet of my Grok is;

   if [program] == "openvpn" {
     grok {  
     match => [ "rest", " (%{USERNAME:openvpn_user}/)?%{IP:openvpn_src_ip}:%{INT:openvpn_scr_port} (\[%{USERNAME:openvpn_user}\])?(peer info: IV\_%{NOTSPACE:openvpn_opt})?%{GREEDYDATA:openvpn_message}" ]
     }
      # OpenVPN Options
      if [openvpn_opt] != '' {
        kv {
         source => 'openvpn_opt'
         prefix => 'openvpn_opt_'
        }
      }

I’m hoping I can aggregate these into a single event. Any ideas how I can achieve this?

[seanziee (Kofi)]

Probably your best bet is the aggregate filter plugin!

[TehMuffinMoo]

Thanks @seanziee,

I’ll have a look this evening and see if I can work it out! I’d probably need to look at timed end events as they all come through in batches at the same time, so a short timeout should be OK. Never used this plugin so hopefully it’s not too difficult!

[seanziee (Kofi)]

There's pretty good documentation on it with 4 distinct examples.

If you have a specific log that always comes last and thus points to an "END" of that string of logs, then example 1 or 2 would work for you.

But looks like most likely example 4 would work for you.