Filebeat for auditd events

[bm_rec]

Hello! Could you help me with my issue?
I need to get security events from auditd. I've updated filebeat from 5.3.2 to 5.6.4 to 6.0.0. Filebeat worked via audisp-simplify. I want to exclude audisp-simplify from the system because Filebeat 6 can work on with auditd log directly.
And now I can't setting up Filebeat. My filebeat.yml:
atlassian-test:~ # grep -nvE '(#|^$)' /etc/filebeat/filebeat.yml
12:filebeat.modules:
29:auth:
30: enabled: true
71:log:
72: enabled: true
76: var.paths: /var/log/audit/audit.log
449:filebeat.config:
450: prospectors:
455: modules:
456: enabled: true
457: path: modules.d/*.yml
458: reload.enabled: true
459: reload.period: 10s
471:tags: ["linux_ib"]
680:output.logstash:
682: enabled: true
684: hosts: ["172.29.39.13:5044"]
1096:setup.template.settings:
1114:setup.kibana:
1171:logging.level: debug
1192:logging.to_files: true
1193:logging.files:
Config is OK, I don't get auditd logs to ElasticSearch and log is here:
https://pastebin.com/24ZKf9Zu.
If I uncomment string 70 - module: auditd, I've got an error:
atlassian-test:~ # filebeat test config
filebeat2017/11/20 14:48:22.319598 beat.go:625: CRIT Exiting: error loading config file: yaml: line 69: did not find expected key
Exiting: error loading config file: yaml: line 69: did not find expected key
Should I use a log prospector? Also I need filtering some useless string. Look at my old config:
atlassian-test:~ # grep -nvE '(#|^$)' /etc/filebeat/filebeat.yml2
12:filebeat.prospectors:
18:- input_type: log
21: paths:
22: - /var/log/audit/audit.log
24: tags: ["linux_ib"]
25: document_type: linuxlogs
26: include_lines: ['terminal', 'identify']
27: exclude_lines: ['cron', 'systemd', 'git', 'java']
28: close_removed: true
29: clean_removed: true
96:output.logstash:
98: hosts: ["172.29.39.13:5044"]
Any suggestions? I don't have auth.log and secure in SLES 12 (why?).

[andrewkroh]

Have you considered Auditbeat? It's currently a beta release.

Docs for the module that collectes audit data: https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-metricset-audit-kernel.html

In comparison to the Filebeat module it's going to be more comprehensive in what it parses because it combines related messages (based on their sequence num) into a single event. It also enriches the events by doing uid/gid conversion.

[bm_rec]

Hmm... Beta is not good idea, because I implement it in production for my client.

[bm_rec]

So, how can I monitor auditd via filebeat? If I run enable command I get an error:

# filebeat modules enable auditd
Module auditd doesn't exists!
# filebeat modules list
Enabled:
Disabled:`

Anyway audit logs appear in ELK but not parsed (its have additional tag beats_input_codec_plain_applied).
And I want to add a prospector for exclude some lines for example.
My config file filebeat.yml now:

  filebeat.modules:
  - module: auditd
    log.enabled: true
  #  filebeat.prospector:
 #- type: log
 #  enabled: true
 #  include_lines: ['terminal', 'identify']
 #  exclude_lines: ['cron', 'systemd', 'git', 'java']
 #  paths:
 #    - /var/log/audit/audit.log
 #  scan_frequency: 5s
 filebeat.config:
   prospectors:
      #enabled: false
      #path: prospectors.d/*.yml
      #reload.enabled: true
      #reload.period: 10s
    modules:
      enabled: true
      path: modules.d/*.yml
      reload.enabled: true
      reload.period: 10s
  tags: ["linux_ib"]
  output.logstash:
    enabled: true
    hosts: ["172.29.39.13:5044"]
    ttl: 90s
  setup.template.settings:
  setup.kibana:
  logging.level: debug
  logging.to_files: true
  logging.files:

[andrewkroh]

All of the Filebeat modules have a dependency on Elasticsearch ingest node so they only work properly when Filebeat outputs directly to Elasticsearch.

@bm_rec

Anyway audit logs appear in ELK but not parsed (its have additional tag beats_input_codec_plain_applied).

And I want to add a prospector for exclude some lines for example.

I believe any of the regular prospector options can be configured for module by putting the option under var (e.g. auditd.log.var.<option_name>).

filebeat.modules:
- module: auditd
  log:
    enabled: true
    var:
      include_lines: ['terminal', 'identify']
      exclude_lines: ['cron', 'systemd', 'git', 'java']