Auditbeat conf.d files

Elastic Stack Beats
1

Auditbeat isn't loading all config files from conf.d, it appears as if the conf files are stomping over each other according to the auditbeat log where the rule count from file 1 is first added and then later deleted when the next cfg file is loaded.

Setup:

pwd
/etc/auditbeat/conf.d

ls
01-first.yml > 42 auditd rules
99-last.yml > 2 auditd rules

auditbeat log:

2018-05-04T10:43:54.141-0500	DEBUG	[modules]	beater/metricbeat.go:80	Register [ModuleFactory:[], MetricSetFactory:[auditd/auditd, file_integrity/file]]
2018-05-04T10:43:54.141-0500	DEBUG	[cfgfile]	cfgfile/reload.go:95	Checking module configs from: /etc/auditbeat/conf.d/*.yml
2018-05-04T10:43:54.141-0500	DEBUG	[cfgfile]	cfgfile/cfgfile.go:143	Load config from file: /etc/auditbeat/conf.d/01-first.yml
2018-05-04T10:43:54.141-0500	DEBUG	[cfgfile]	cfgfile/cfgfile.go:143	Load config from file: /etc/auditbeat/conf.d/99-last.yml
2018-05-04T10:43:54.142-0500	DEBUG	[cfgfile]	cfgfile/reload.go:109	Number of module configs found: 3
2018-05-04T10:43:54.144-0500	INFO	[auditd]	auditd/audit_linux.go:65	auditd module is running as euid=0 on kernel=3.10.0-693.21.1.el7.x86_64
2018-05-04T10:43:54.195-0500	INFO	[auditd]	auditd/audit_linux.go:88	socket_type=unicast will be used.
2018-05-04T10:43:54.196-0500	INFO	[auditd]	auditd/audit_linux.go:65	auditd module is running as euid=0 on kernel=3.10.0-693.21.1.el7.x86_64
2018-05-04T10:43:54.247-0500	INFO	[auditd]	auditd/audit_linux.go:88	socket_type=unicast will be used.
2018-05-04T10:43:54.247-0500	INFO	cfgfile/reload.go:127	Config reloader started
2018-05-04T10:43:54.247-0500	DEBUG	[cfgfile]	cfgfile/reload.go:151	Scan for new config files
2018-05-04T10:43:54.247-0500	DEBUG	[cfgfile]	cfgfile/cfgfile.go:143	Load config from file: /etc/auditbeat/conf.d/01-first.yml
2018-05-04T10:43:54.247-0500	DEBUG	[cfgfile]	cfgfile/cfgfile.go:143	Load config from file: /etc/auditbeat/conf.d/99-last.yml
2018-05-04T10:43:54.248-0500	DEBUG	[cfgfile]	cfgfile/reload.go:170	Number of module configs found: 3
2018-05-04T10:43:54.248-0500	DEBUG	[cfgfile]	cfgfile/reload.go:198	Remove module from stoplist: 13250497449215009278
2018-05-04T10:43:54.248-0500	DEBUG	[cfgfile]	cfgfile/reload.go:203	Add module to startlist: 13250497449215009278
2018-05-04T10:43:54.249-0500	INFO	[auditd]	auditd/audit_linux.go:65	auditd module is running as euid=0 on kernel=3.10.0-693.21.1.el7.x86_64
2018-05-04T10:43:54.302-0500	INFO	[auditd]	auditd/audit_linux.go:88	socket_type=unicast will be used.
2018-05-04T10:43:54.302-0500	DEBUG	[cfgfile]	cfgfile/reload.go:198	Remove module from stoplist: 3196340903787212030
2018-05-04T10:43:54.302-0500	DEBUG	[cfgfile]	cfgfile/reload.go:203	Add module to startlist: 3196340903787212030
2018-05-04T10:43:54.303-0500	DEBUG	[cfgfile]	cfgfile/reload.go:198	Remove module from stoplist: 8453206759139377238
2018-05-04T10:43:54.303-0500	DEBUG	[cfgfile]	cfgfile/reload.go:203	Add module to startlist: 8453206759139377238
2018-05-04T10:43:54.303-0500	INFO	[auditd]	auditd/audit_linux.go:65	auditd module is running as euid=0 on kernel=3.10.0-693.21.1.el7.x86_64
2018-05-04T10:43:54.354-0500	INFO	[auditd]	auditd/audit_linux.go:88	socket_type=unicast will be used.
2018-05-04T10:43:54.354-0500	INFO	cfgfile/reload.go:258	Starting 3 runners ...
2018-05-04T10:43:54.354-0500	DEBUG	[cfgfile]	cfgfile/reload.go:265	New runner started: 13250497449215009278
2018-05-04T10:43:54.354-0500	DEBUG	[cfgfile]	cfgfile/reload.go:265	New runner started: 3196340903787212030
2018-05-04T10:43:54.354-0500	DEBUG	[cfgfile]	cfgfile/reload.go:265	New runner started: 8453206759139377238
2018-05-04T10:43:54.354-0500	INFO	cfgfile/reload.go:219	Loading of config files completed.
2018-05-04T10:43:55.614-0500	INFO	[auditd]	auditd/audit_linux.go:158	Deleted 2 pre-existing audit rules.
2018-05-04T10:43:55.621-0500	INFO	[auditd]	auditd/audit_linux.go:171	Successfully added 42 of 42 audit rules.
2018-05-04T10:43:55.621-0500	INFO	[auditd]	auditd/audit_linux.go:192	audit status from kernel at start	{"audit_status": {"Mask":0,"Enabled":1,"Failure":0,"PID":0,"RateLimit":0,"BacklogLimit":8192,"Lost":692,"Backlog":0,"FeatureBitmap":61,"BacklogWaitTime":0}}
2018-05-04T10:43:55.633-0500	INFO	[auditd]	auditd/audit_linux.go:158	Deleted 2 pre-existing audit rules.
2018-05-04T10:43:55.638-0500	INFO	[auditd]	auditd/audit_linux.go:171	Successfully added 2 of 2 audit rules.

According to the comments and online docs each file in the /conf.d/ directory should be loaded if it has the right syntax but it seems that you cannot have two cfg files with the same module. Has anyone else been able to get this to work as documented?

2

Modified the two config files to do a file watch of a specific file in each config for testing, rule list:

01-first.yml

- module: auditd
  audit_rules: |
    -w /opt/first_config -p wa -k first_config

99-last.yml

- module: auditd
  audit_rules: |
    -w /opt/last_config -p wa -k last_config

When I edit the files in /opt after restarting audit beat, only rules from the first config seem to be working. Looking at the output files it looks like rules are loaded from last config first and then deleted when rules are loaded from the first config.

3

The auditd module doesn't fully support multi-instantiation. Each instance of the module expects to be in full control over the kernel audit subsystem. So you should only have a single instance of this module in your config.

Are you looking to separate your rules into different files like you do with auditd in /etc/audit/audit.rules.d? I've been thinking to add a config option for reading the rules from a directory like that.

1 Like
4

Yes, I may have interpreted the comments or docs incorrectly but I thought that was the point of enabling the /conf.d/ config loading like logstash does.

My goal is to maintain multiple configs in /conf.d/* one per team that has rule requirements without having 1 massive rule file. I have different teams dependent on audit rules like compliance, security and app teams. The plan was to have a conf file per team where they can configure their own rules. I'm not sure what happens for performance when I tell auditbeat to watch a file that doesn't exist but if I were to maintain 1 large config file, I'll be looking for hundreds of files that don't exist due to the various app teams that would have their own file watches. If I were to do this same thing with auditd it would work because any file or path that didn't already exist would just error out when trying to add that specific rule.

It would also be nice if there was an option to dump the list of current rules that are running, since the logs showed that rules were deleted and some where added, but I had no idea what was actually loaded after startup, similar to how you can run auditctl -l to dump the currently loaded rules. I could probably comb thorough the output of auditbeat to see what rules were added and if they were eventually deleted to build that list, but a local get status command would have been helpful or occasional printing them with a debug option.

5

That's all good feedback. It sounds like three separate changes to Auditbeat:

  • Allow audit rules to be defined across multiple files so different teams can define rules.
  • Add a CLI flag for dumping the kernel's current audit rules
  • Skip adding audit file watches for paths that don't exist (Is this what you meant?)

I'll create an enhancement request for each of them mid-next week (off for a few days).

6

Thanks,

For file_integrity module, I'd suggest skipping paths and files that don't already exist with a log entry that says why a rule was skipped. Maybe a simple config setting of skip_invalid = true to enable that feature would work best and similar to how you turn on recursive.

For the audit_rules module, I think it already skips invalid paths without breaking but I haven't spent a ton of time testing this along with performance, auditd can be configured to skip invalid rules and continue loading the rest of the rules, it can also be configured to bail out and stop loading rules (please don't do this).

The last missing feature might be a file_integrity re-scan interval, I've been testing this with modifications to /etc/sudo.conf and I can't seem to trigger the FIM event unless I restart auditbeat. The docs say that it should send events when files are changed (created, deleted, updated) but this doesn't seem to be working for small changes. Is there a syscall required to trigger a rescan of the file? or is there some sort of threshold (ttl) that can be modified to force a rescan? Maybe the setting scan_at_start: true really only scans at startup only and I misinterpreted that setting.

update: I striped out all syscall rules to test the file_integrity module and now everything seems to be working as expected and in real-time. I'll troubleshoot some more to see if something was delaying the write of the events, but the FIM module seems to be working fine.

7

It uses inotify to watch for changes to the file in near realtime. At startup it will add the inotify watches and then do an initial scan where it manually traverses all the files to check for changes since it was last run. So when you change the file and Auditbeat is running it should be receiving an inotify event saying something changed, go look.

If you enable debug for the the file integrity module (by adding these settings to the config logging.level: debug and logging.selectors: [file_integrity]) you should see in the log that an inotify event was received then Auditbeat will send an event if something changed.

1 Like
8

Would you be able to add another issue related to links (symbolic and hard)

Link example:

ln -s /etc/ssh/sshd_config /opt/file_integrity/symbolic_sshd_config
ln /etc/ssh/sshd_config /opt/file_integrity/hard_sshd_config

Config Example 1 (recursive):

- module: file_integrity
  paths:
  - /opt/file_integrity
  recursive: true

Using this config auditbeat will log if the hard link file under /opt/file_integrity is changed, however nothing is caught if the original source (/etc/ssh/sshd_config) is modified, nothing is logged if you edit the file through the symlink, when restarted the re-scan will find that the hardlink file was changed and log an event.

Config Example 2 (not recursive)

- module: file_integrity
  paths:
  - /opt/file_integrity/symbolic_sshd_config
  - /opt/file_integrity/hard_sshd_config

Changes to the files through links or the target files are caught with fsnotify and events are generated as expected. The event data seems to only log the hard link name, but modifications to the source /etc/ file are logged as hard link name. Someone could bypass auditbeat syntax, filters, or downstream correlation tools if they make a hardlink of a file before making malicious changes. Example; if your security team is only looking for changes to static list of critical files, this would change the filename in the event possibly bypassing security tools.

Config Example 3 (Source file non recursive)

- module: file_integrity
  paths:
  - /etc/ssh/sshd_config

Changes to the file through links and directly are logged as expected. No issues.

Config Example 4 (single file recursive)

- module: file_integrity
  paths:
  - /opt/file_integrity/hard_sshd_config
  recursive: true

Using this config, auditbeat doesn't get any fsnotify events in the debug output but will still notify of changes when auditbeat is restarted.

Something seems to be impacting the fsnotify alert when recursive is on. Maybe its possible that hard links are just going to be an issue here and symlinks aren't followed with recursive on, if this is intended, please add link details in the documentation.

9

I created two issues so far. I need to do a bit of testing with the symlinks before I create that issue.

When the file_integrity module skips a path due to an error it should be logging a warning like

2018-05-16T00:16:22.604Z	WARN	[file_integrity]	file_integrity/eventreader_fsnotify.go:46	Failed to add watch	{"file_path": "/does_not_exist", "error": "1 error: recursion into dir '/does_not_exist' failed: no such file or directory"}