Hi all,
I found that the issue discussed in:
is still relevant in 2019 using the default audit output in JSON (ref: log4j2.properties for Elasticsearch 7.x) which is:
appender.audit_rolling.layout.pattern = {\
"@timestamp":"%d{ISO8601}"\
%varsNotEmpty{, "node.name":"%enc{%map{node.name}}{JSON}"}\
%varsNotEmpty{, "node.id":"%enc{%map{node.id}}{JSON}"}\
%varsNotEmpty{, "host.name":"%enc{%map{host.name}}{JSON}"}\
%varsNotEmpty{, "host.ip":"%enc{%map{host.ip}}{JSON}"}\
%varsNotEmpty{, "event.type":"%enc{%map{event.type}}{JSON}"}\
%varsNotEmpty{, "event.action":"%enc{%map{event.action}}{JSON}"}\
%varsNotEmpty{, "user.name":"%enc{%map{user.name}}{JSON}"}\
%varsNotEmpty{, "user.run_by.name":"%enc{%map{user.run_by.name}}{JSON}"}\
%varsNotEmpty{, "user.run_as.name":"%enc{%map{user.run_as.name}}{JSON}"}\
%varsNotEmpty{, "user.realm":"%enc{%map{user.realm}}{JSON}"}\
%varsNotEmpty{, "user.run_by.realm":"%enc{%map{user.run_by.realm}}{JSON}"}\
%varsNotEmpty{, "user.run_as.realm":"%enc{%map{user.run_as.realm}}{JSON}"}\
%varsNotEmpty{, "user.roles":%map{user.roles}}\
%varsNotEmpty{, "origin.type":"%enc{%map{origin.type}}{JSON}"}\
%varsNotEmpty{, "origin.address":"%enc{%map{origin.address}}{JSON}"}\
%varsNotEmpty{, "realm":"%enc{%map{realm}}{JSON}"}\
%varsNotEmpty{, "url.path":"%enc{%map{url.path}}{JSON}"}\
%varsNotEmpty{, "url.query":"%enc{%map{url.query}}{JSON}"}\
%varsNotEmpty{, "request.method":"%enc{%map{request.method}}{JSON}"}\
%varsNotEmpty{, "request.body":"%enc{%map{request.body}}{JSON}"}\
%varsNotEmpty{, "request.id":"%enc{%map{request.id}}{JSON}"}\
%varsNotEmpty{, "action":"%enc{%map{action}}{JSON}"}\
%varsNotEmpty{, "request.name":"%enc{%map{request.name}}{JSON}"}\
%varsNotEmpty{, "indices":%map{indices}}\
%varsNotEmpty{, "opaque_id":"%enc{%map{opaque_id}}{JSON}"}\
%varsNotEmpty{, "x_forwarded_for":"%enc{%map{x_forwarded_for}}{JSON}"}\
%varsNotEmpty{, "transport.profile":"%enc{%map{transport.profile}}{JSON}"}\
%varsNotEmpty{, "rule":"%enc{%map{rule}}{JSON}"}\
%varsNotEmpty{, "event.category":"%enc{%map{event.category}}{JSON}"}\
}%n
will generate a document like this:
{"@timestamp":"2019-08-28T07:20:26,164", "node.name":"dev-data-4-instance-1", "node.id":"ZJl7IaozQNSe3bvvfQk2iw", "event.type":"transport", " **event.action** ":"access_granted", " user.name ":"logstash", "user.realm":"native", "user.roles":["logstash"], " origin.type ":"local_node", "origin.address":"10.66.41.223:9301", "request.id":"0OHMnG9GSRWzcAwrCgFDNg", "action":"indices:data/write/bulk[s][r]", "request.name":"BulkShardRequest", "indices":["myindex-7d-2019.08.28"]}
Which is impossible to filter with a valid processor config (tested on 5.x and 6.x) like this:
processors:
- drop_event:
when:
- equals:
user.name: logstash
or
processors:
- drop_event.when.equals.user.name: 'logstash'
In the end I filtered at the source by putting in elasticsearch.yml:
xpack.security.audit.logfile.events.ignore_filters:
filter_unwanted_users:
users = [ ...]
But that of course will need a rolling restart and it's kind of less nice than just use filebeat (of course it has the advantage of not even writing to the disk).