Drop event processor not working on Filebeat

Elastic Stack Beats
1

Hello,

I'm trying to create a drop_event processor to only allow elasticsearch audit logs which have a request.name = "AuthenticateRequest". Clearly my process it not working as all events are not matching and everything is being dropped.

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - drop_event:
      when:
        not:
          request.name:"AuthenticateRequest"

Here is a sample of the data being logged to disk with the key value pair matching the log entries but they do not log to Elasticsearch.

{"type":"audit", "timestamp":"2023-07-31T14:58:05,675-0400", "cluster.uuid":"mmsQadMlRySqZ5FVIVZu_w", "node.id":"ItmwGws8Qu-xismk7PwdMA", "event.type":"transport", "event.action":"access_granted", "authentication.type":"REALM", "user.name":"monitor", "user.realm":"native1", "user.roles":["kibana_monitor","elasticsearch_monitor"], "origin.type":"rest", "origin.address":"x.x.x.x:47702", "request.id":"Thfm_mHwTkKA25MHTatgag", "action":"cluster:admin/xpack/security/user/authenticate", "request.name":"AuthenticateRequest", "opaque_id":"cc567b01-8f8b-452c-b251-e41d4a061b84", "trace.id":"d6f7a0878ab69b081e0f9a9848c9e6a3"}
{"type":"audit", "timestamp":"2023-07-31T14:58:05,681-0400", "cluster.uuid":"mmsQadMlRySqZ5FVIVZu_w", "node.id":"ItmwGws8Qu-xismk7PwdMA", "event.type":"transport", "event.action":"access_granted", "authentication.type":"REALM", "user.name":"monitor", "user.realm":"native1", "user.roles":["kibana_monitor","elasticsearch_monitor"], "origin.type":"rest", "origin.address":"x.x.x.x:52116", "request.id":"enO8cVK4SiqsnCP27KsM7Q", "action":"cluster:admin/xpack/security/user/authenticate", "request.name":"AuthenticateRequest", "opaque_id":"ab1e85cc-8dfc-4eb3-8aaa-03ffae7fa30c", "trace.id":"d6ac44251f290be0ec069686a55b2878"}

If I change the logic to match a specific key value pair, it does not work either using the Elasticsearch audit logs.

I'd appreciate any help. Thank you!

2

Hey @JeremyP,

I think you are missing the comparison function equals. Try with this:

  - drop_event:
      when:
        not:
          equals:
            request.name: "AuthenticateRequest"

I wonder in any case why Filebeat was not complaining about invalid condition :thinking:

3

Thank you. I've made the modification and no filebeat events are streaming into Elasticsearch.... it's dropping everything.

  - drop_event:
      when:
        not:
          equals:
            request.name:"AuthenticateRequest"

I'd expect both these logs to show up in the cluster, but they do not. If I remove the drop_event filter, everything streams as normal.

{"type":"audit", "timestamp":"2023-08-01T10:41:15,679-0400", "cluster.uuid":"mmsQadMlRySqZ5FVIVZu_w", "node.id":"ItmwGws8Qu-xismk7PwdMA", "event.type":"transport", "event.action":"access_granted
", "authentication.type":"REALM", "user.name":"monitor", "user.realm":"native1", "user.roles":["kibana_monitor","elasticsearch_monitor"], "origin.type":"rest", "origin.address":"x.x.x.x:413
80", "request.id":"OWY3qrrCTlud4MmeMhlNYg", "action":"cluster:admin/xpack/security/user/authenticate", "request.name":"AuthenticateRequest", "opaque_id":"6a799602-ebdb-45a1-9f0f-2ace874753d3", "
trace.id":"18074220133e12038d621a4ba73de94f"}
{"type":"audit", "timestamp":"2023-08-01T10:41:15,682-0400", "cluster.uuid":"mmsQadMlRySqZ5FVIVZu_w", "node.id":"ItmwGws8Qu-xismk7PwdMA", "event.type":"transport", "event.action":"access_granted
", "authentication.type":"REALM", "user.name":"monitor", "user.realm":"native1", "user.roles":["kibana_monitor","elasticsearch_monitor"], "origin.type":"rest", "origin.address":"x.x.x.x:534
38", "request.id":"gpF-1K6dRN66quniH0Lf7Q", "action":"cluster:admin/xpack/security/user/authenticate", "request.name":"AuthenticateRequest", "opaque_id":"885d8053-fb89-4aa7-b8e8-288422a40069", "
trace.id":"87b356c74866e5a1245af6819c4116f7"}
4

Are you parsing these logs locally or using ingest pipelines? The request.name field should exist locally.

5

Elastic support assisted and they suggested adding this process prior to the drop_event. According to them, the data has not been decoded yet by filebeat prior to the drop_event.

  - decode_json_fields: 
      fields: ["message"]
      target:

This has resolved the issue for me.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.