Logstash aggregate filter on auditd logs

[Nadavr (Nadav)]

Hi Everyone,
I'm trying to parse Linux audit.log files with logstash.
Every execution of shell command (in this example: tail /etc/hosts) generates 5 different event types,
relating to the same action, having the same timestamp and event ID:

type=SYSCALL msg=audit(1503851920.016:3077): arch=c000003e syscall=59 success=yes exit=0 a0=1950ec0 a1=196b4e0 a2=194e8a0 a3=7ffee54b5870 items=2 ppid=9493 pid=9509 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=132 comm="tail" exe="/usr/bin/tail" key=(null)
type=EXECVE msg=audit(1503851920.016:3077): argc=2 a0="tail" a1="/etc/hosts"
type=CWD msg=audit(1503851920.016:3077):  cwd="/root"
type=PATH msg=audit(1503851920.016:3077): item=0 name="/usr/bin/tail" inode=16780926 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL
type=PATH msg=audit(1503851920.016:3077): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=33597599 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL

The pattern is basically:

type=%{DATA:audit_type}\smsg=audit\(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}\):.*?( msg=\'(?<sub_msg>.*?)\')?$

I wish to aggregate the logs based on timestamp and event ID and I'm really having difficulties doing that.
I'd be glad to get a push in the right direction.

Thanks in advnace.

[pjanzen (Paul Janzen)]

So if I understand correctly you want to combine those 5 events to 1 event and store that in elasticsearch?
If so, this is not going to be easy I think, as logstash sees 5 event and as far as I know you cannot "store" events.

That said, you might be able to alther the logging of auditd so it produces 1 line that stores all that data or create a syslog-ng / rsyslog template that does it for you.

Other question, why not store them individually?

Paul.