Massive /var/log/audit logging

Elastic Stack Elasticsearch
1

We configure our nodes with basic Ubuntu CIS (https://benchmarks.cisecurity.org/tools2/linux/CIS_Ubuntu_14.04_LTS_Server_Benchmark_v1.0.0.pdf).

On our elasticsearch data nodes this results in an amazing number of audit logs. So much that it fills our 24G log volumes in just a few hours. The logs look like

type=SYSCALL msg=audit(1469521785.213:19745470): arch=c000003e syscall=87 success=yes exit=0 a0=7f5030092480 a1=7f4d30c92530 a2=7f5030092480 a3=7f5039f3ebc1 items=2 ppid=1 pid=17326 auid=1003 uid=20000 gid=20000 euid=20000 suid=20000 fsuid=20000 egid=20000 sgid=20000 fsgid=20000 tty=(none) ses=2 comm="java" exe="/usr/lib/jvm/java-8-oracle/jre/bin/java" key="delete"
type=CWD msg=audit(1469521785.213:19745470):  cwd="/usr/share/elasticsearch"
type=PATH msg=audit(1469521785.213:19745470): item=0 name="/data/es/staging/nodes/0/indices/connect-mtconnectdataitems-201607/2/index/" inode=7080182 dev=ca:50 mode=040755 ouid=20000 ogid=20000 rdev=00:00 nametype=PARENT
type=PATH msg=audit(1469521785.213:19745470): item=1 name="/data/es/staging/nodes/0/indices/connect-mtconnectdataitems-201607/2/index/_4ol2.fnm" inode=7079803 dev=ca:50 mode=0100644 ouid=20000 ogid=20000 rdev=00:00 nametype=DELETE
type=SYSCALL msg=audit(1469521785.213:19745471): arch=c000003e syscall=87 success=yes exit=0 a0=7f5030092480 a1=7f4d30c92530 a2=7f5030092480 a3=7f5039f3ebc1 items=2 ppid=1 pid=17326 auid=1003 uid=20000 gid=20000 euid=20000 suid=20000 fsuid=20000 egid=20000 sgid=20000 fsgid=20000 tty=(none) ses=2 comm="java" exe="/usr/lib/jvm/java-8-oracle/jre/bin/java" key="delete"
type=CWD msg=audit(1469521785.213:19745471):  cwd="/usr/share/elasticsearch"
type=PATH msg=audit(1469521785.213:19745471): item=0 name="/data/es/staging/nodes/0/indices/connect-mtconnectdataitems-201607/2/index/" inode=7080182 dev=ca:50 mode=040755 ouid=20000 ogid=20000 rdev=00:00 nametype=PARENT
type=PATH msg=audit(1469521785.213:19745471): item=1 name="/data/es/staging/nodes/0/indices/connect-mtconnectdataitems-201607/2/index/_4ol2.nvd" inode=7078132 dev=ca:50 mode=0100644 ouid=20000 ogid=20000 rdev=00:00 nametype=DELETE

Is this normal. Should we expect ES to be creating and deleting files so rapidly?

2

This seems not as an Elasticsearch-related question - but yes, Elasticsearch rapidly creates and deletes tens of thousands of files.

You are running Ubuntu "CIS", which I identify as "Center for Internet Security" (whatever that means), so I guess you are running a Linux kernel with fully enabled audit logs, which is quite uncommon.

You can switch kernel auditing temporarily off by

/usr/sbin/auditctl -e 0

or you can switch it off permanently by configuring your Linux kernel boot line b ythe parameter audit=0, see cat /proc/cmdline

Other configuration about managing auditing configuration and logs is available in /etc/audit/auditd.conf

3

Thanks @jprante. I don't really want to disable the directories that logstash writes to via auditd.rules.

I really wanted to know if generating 1GB/hour of logs like the above we might indicate that we have some misconfiguration that is causing ES to thrash.