Filebeat and fields with dots in name

Elastic Stack Beats
1

Hi all,

I found that the issue discussed in:

is still relevant in 2019 using the default audit output in JSON (ref: log4j2.properties for Elasticsearch 7.x) which is:

appender.audit_rolling.layout.pattern = {\
                "@timestamp":"%d{ISO8601}"\
                %varsNotEmpty{, "node.name":"%enc{%map{node.name}}{JSON}"}\
                %varsNotEmpty{, "node.id":"%enc{%map{node.id}}{JSON}"}\
                %varsNotEmpty{, "host.name":"%enc{%map{host.name}}{JSON}"}\
                %varsNotEmpty{, "host.ip":"%enc{%map{host.ip}}{JSON}"}\
                %varsNotEmpty{, "event.type":"%enc{%map{event.type}}{JSON}"}\
                %varsNotEmpty{, "event.action":"%enc{%map{event.action}}{JSON}"}\
                %varsNotEmpty{, "user.name":"%enc{%map{user.name}}{JSON}"}\
                %varsNotEmpty{, "user.run_by.name":"%enc{%map{user.run_by.name}}{JSON}"}\
                %varsNotEmpty{, "user.run_as.name":"%enc{%map{user.run_as.name}}{JSON}"}\
                %varsNotEmpty{, "user.realm":"%enc{%map{user.realm}}{JSON}"}\
                %varsNotEmpty{, "user.run_by.realm":"%enc{%map{user.run_by.realm}}{JSON}"}\
                %varsNotEmpty{, "user.run_as.realm":"%enc{%map{user.run_as.realm}}{JSON}"}\
                %varsNotEmpty{, "user.roles":%map{user.roles}}\
                %varsNotEmpty{, "origin.type":"%enc{%map{origin.type}}{JSON}"}\
                %varsNotEmpty{, "origin.address":"%enc{%map{origin.address}}{JSON}"}\
                %varsNotEmpty{, "realm":"%enc{%map{realm}}{JSON}"}\
                %varsNotEmpty{, "url.path":"%enc{%map{url.path}}{JSON}"}\
                %varsNotEmpty{, "url.query":"%enc{%map{url.query}}{JSON}"}\
                %varsNotEmpty{, "request.method":"%enc{%map{request.method}}{JSON}"}\
                %varsNotEmpty{, "request.body":"%enc{%map{request.body}}{JSON}"}\
                %varsNotEmpty{, "request.id":"%enc{%map{request.id}}{JSON}"}\
                %varsNotEmpty{, "action":"%enc{%map{action}}{JSON}"}\
                %varsNotEmpty{, "request.name":"%enc{%map{request.name}}{JSON}"}\
                %varsNotEmpty{, "indices":%map{indices}}\
                %varsNotEmpty{, "opaque_id":"%enc{%map{opaque_id}}{JSON}"}\
                %varsNotEmpty{, "x_forwarded_for":"%enc{%map{x_forwarded_for}}{JSON}"}\
                %varsNotEmpty{, "transport.profile":"%enc{%map{transport.profile}}{JSON}"}\
                %varsNotEmpty{, "rule":"%enc{%map{rule}}{JSON}"}\
                %varsNotEmpty{, "event.category":"%enc{%map{event.category}}{JSON}"}\
                }%n

will generate a document like this:

{"@timestamp":"2019-08-28T07:20:26,164", "node.name":"dev-data-4-instance-1", "node.id":"ZJl7IaozQNSe3bvvfQk2iw", "event.type":"transport", " **event.action** ":"access_granted", " user.name ":"logstash", "user.realm":"native", "user.roles":["logstash"], " origin.type ":"local_node", "origin.address":"10.66.41.223:9301", "request.id":"0OHMnG9GSRWzcAwrCgFDNg", "action":"indices:data/write/bulk[s][r]", "request.name":"BulkShardRequest", "indices":["myindex-7d-2019.08.28"]}

Which is impossible to filter with a valid processor config (tested on 5.x and 6.x) like this:

processors:
      - drop_event:
        when:
          - equals:
            user.name: logstash

or

processors:

- drop_event.when.equals.user.name: 'logstash'

In the end I filtered at the source by putting in elasticsearch.yml:

xpack.security.audit.logfile.events.ignore_filters:
        filter_unwanted_users:
          users = [ ...]

But that of course will need a rolling restart and it's kind of less nice than just use filebeat (of course it has the advantage of not even writing to the disk).

2

Hi,

This syntax seemed to work for me in a quick test:

 - drop_event:
     when.equals:
       "user.name": logstash

The quotes make the YAML parser pass that as a single key. Let me know if that doesn't work for you.

1 Like
3

Thanks for the reply, I'm pretty sure I tried but it's worth retrying again, maybe I used single quotes? Can't remember right now. I'll let you know asap.

4

Well, this doesn't work for me (filebeat-5.6.11):

---
filebeat:
  prospectors:
    - input_type: log
      paths:
        - /var/log/elasticsearch/instance-1/*_audit.json
      encoding: plain
      fields_under_root: false
      document_type: elastic-audit
      scan_frequency: 10s
      harvester_buffer_size: 16384
      max_bytes: 10485760
      processors:
      - drop_event:
        when.equals:
            "user.name": kibana
5

It might also be something related to the puppet module and/or the yaml syntax.
The important thing is that the syntax with double quote is supported and working for you (and possibly for everyone getting here for the same problem).
I'm not sure right now that this is a widespread problem.

6

We are following up this behavior on a support ticket, will update the discussion with our findings.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.