I was using auditbeat to track audit events. However, if it goes down for a long time, I lose events, as I mentioned in Auditd events created while Auditbeat is down. Due to that, my team wants to use log files to store events using auditd and move the critical point of failure from auditbeat to auditd, which belongs to the CentOS distribution we are using and is outside of the scope of our tracking solution.
I'm checking if I could use filebeat to parse auditd log file. I want to get the multiline events, transform it to a single event in JSON format and output it to Elasticsearch. auditd log file example:
type=LOGIN msg=audit(1613845201.335:6537): pid=32161 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=350 res=1
type=USER_START msg=audit(1613845201.349:6538): pid=32161 uid=0 auid=0 ses=350 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_REFR msg=audit(1613845201.349:6539): pid=32161 uid=0 auid=0 ses=350 msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_DISP msg=audit(1613845201.356:6540): pid=32161 uid=0 auid=0 ses=350 msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_END msg=audit(1613845201.357:6541): pid=32161 uid=0 auid=0 ses=350 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=SYSCALL msg=audit(1613845287.701:6542): arch=c000003e syscall=2 success=yes exit=17 a0=55f360492dc0 a1=20000 a2=1e4 a3=4000 items=1 ppid=1792 pid=2435 auid=4294967295 uid=201327710 gid=0 euid=201327710 suid=0 fsuid=201327710 egid=201327708 sgid=0 fsgid=201327708 tty=(none) ses=4294967295 comm="smbd" exe="/usr/sbin/smbd" key=(null)
type=CWD msg=audit(1613845287.701:6542): cwd="/sasdata"
type=PATH msg=audit(1613845287.701:6542): item=0 name="." inode=100916940 dev=08:04 mode=040775 ouid=0 ogid=201327708 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PROCTITLE msg=audit(1613845287.701:6542): proctitle="/usr/sbin/smbd"
type=SYSCALL msg=audit(1613845287.702:6543): arch=c000003e syscall=2 success=yes exit=17 a0=55f360492dc0 a1=30000 a2=0 a3=4000 items=1 ppid=1792 pid=2435 auid=4294967295 uid=201327710 gid=0 euid=201327710 suid=0 fsuid=201327710 egid=201327708 sgid=0 fsgid=201327708 tty=(none) ses=4294967295 comm="smbd" exe="/usr/sbin/smbd" key=(null)
type=CWD msg=audit(1613845287.702:6543): cwd="/sasdata"
type=PATH msg=audit(1613845287.702:6543): item=0 name="." inode=100916940 dev=08:04 mode=040775 ouid=0 ogid=201327708 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PROCTITLE msg=audit(1613845287.702:6543): proctitle="/usr/sbin/smbd"
type=NETFILTER_CFG msg=audit(1613845330.926:6544): table=nat family=2 entries=0
type=NETFILTER_CFG msg=audit(1613845330.926:6544): table=filter family=2 entries=0
type=NETFILTER_CFG msg=audit(1613845330.926:6544): table=mangle family=2 entries=0
The simplest algorithm would be to group all audit events which have the same timestamp and audit event ID (SYSCALL, CWD, PATH and PROCTITLE with same timestamp 1613845287.701 and audit event ID 6542) in a single event. I have checked Manage multiline messages | Filebeat Reference [7.11] | Elastic , but AFAIK I cannot do that with available regular expressions features. Can I? If not, would it be easy to change Filebeat code to support group log lines with a common ID in a single event?