Docker implementation - Filebeats 7.17.2 - drop_events not working

Elastic Stack Beats
,
1

I am trying to setup filebeats to monitor some docker containers - I am using autodiscover, but I don't want to be logging anything from my stack (Elastic, Kibana, Filebeats itself). I have tried to use drop_events but it's just not working:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      processors:
        - drop_event.when:
            - equals.docker.container.name: kibana
            - contains.docker.image.name: kibana
            - equals.docker.container.name: elasticsearch
            - contains.docker.image.name: elasticsearch
            - equals.docker.container.name: filebeat
            - contains.docker.image.name: filebeat

I also tried:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      processors:
        - drop_event.when.not:
            - equals.docker.container.name: httpd

Same issue.

I've found a couple of old posts which are related such as this one here - But they all seem to have gone un-answered.

Any help would be appreciated!

2

Hi @amills157 Welcome to the community.

First observation If you don't want to keep logs from your ELK stack then why are you using the not basically that will drop everything except for the stack logs.

Perhaps I am missing something.

Also if you want that to be an or you need use an or

I don't believe your syntax is correct it is very specific.

Also, the drop only work on fields that already exist in the message, not once it will be created if you're using a module or a pipeline.

The container names or images I suspect should be available for you to use but you need to use the correct field names and values.

What I would do is write a single "positive" logic drop and get that to work.. then add with the or

- drop_event:
    when:
      field name : value
3

Hi @stephenb ,

No apologies you're right the .not in the first example is a typo from where I copy pasta'd the httpd version of the config. I'll amend that now (the actual config being used didn't have that not and I have double checked / tested that this morning and I'm still seeing logs from KIbana etc) .

As far as I can work out though the httpd config should be dropping everything but traffic from the httpd container (which it isn't).

The container name in this instance is correct as the container is called httpd so I'm not sure I understand your point in regards to the field name : value as that should match what I am trying in this instance.

I did previously try:

- drop_event:
    - when:
        - contains:
            docker.image.name: kibana

Same issue though I kept seeing kibana logs appearing

4

First you keep putting in - when they are not needed / not correct.

So here is a sample I will use... this is the JSON message in discover

    "container": {
      "image": {
        "name": "gcr.io/google-samples/istio/helloserver:v0.0.1"
      },
      "runtime": "containerd",
      "id": "c0b67738739fc5877596a3f478002ba1f282c2266ea33df6f54f90243471a94b"
    },

Screen Shot 2022-05-01 at 8.33.47 AM
Screen Shot 2022-05-01 at 8.33.47 AM2580×380 39.1 KB

Say I want to drop this then my drop will be ... and this works I tested it.
NOTE 2 spaces Indent no extra -

      - drop_event:
          when:
            contains:
              container.image.name: helloserver

This also works which I actually like better

      - drop_event.when.contains.container.image.name: helloserver

Perhaps show us the actual field name and values .... and we can help

5

Hi @stephenb,

Apologies for any syntax issues there - The top two / originally posted configs had been modelled on those in this post which made use of the - as mine does.

I have amended to follow your example:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      processors:
        - drop_event.when.contains.container.image.name: kibana

However as shown here that hasn't worked either (kibana feed following the most recent config edit)

Screenshot at 2022-05-01 17-19-48
Screenshot at 2022-05-01 17-19-48964×688 88.7 KB

You can see that the image / container names should be matching (or not matching for the httpd rule) but I'm getting all logs regardless of the config used.

6

Please show your entire filebeat conf not just a snippet.

Also try this move the processor to the left 2 spaces to be at the same level of the - type

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
    processors:
      - drop_event.when.contains.container.image.name: kibana

Technically processors can go at the top level

7

Note if you use the provided sample here

The processors are at the Top Level and will act on every message.

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
  - add_cloud_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

so your processor would go

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
  - add_cloud_metadata: ~
  - drop_event.when.contains.container.image.name: kibana  <!--- Here


output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'
8

Hi @stephenb,

Moving the processors to the top level has fixed it. I had initially moved the processors inside the autodiscover after seeing previous posts with that (and no comments to suggest it was wrong). Lesson learnt there and thanks for your help - To help anyone who has this issue the full config used is:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
  - drop_event.when.contains.container.image.name: kibana
  - drop_event.when.contains.container.image.name: elasticsearch
  ...
  - drop_event.when.contains.container.image.name: filebeat
 

output.elasticsearch:
...
1 Like
9

I think it will also work at the same level of the - type ... but I think you had it too deep.
Good ... glad we could help!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.