Filebeat conditional on annotation

Elastic Stack Beats
1

Greetings! I am having trouble using a kubernetes pod annotation in a filebeat autodiscover conditional.

I see that metricbeat's autodiscover kubernetes example has include_annotations then a condition that looks at an annotation. So far I have not been able to get this to work with filebeat.

Filebeat has a similar example which does not include include_annotations. Should the same functionality be possible in filebeat? Also, is it possible to debug conditional configurations? I would like to be able to see what fields are available and their values so I can know what conditions are possible. Thank you!

2

Hi @Jim_Sheldon,

include_annotations works in the same way for both Metricbeat and Filebeat Kubernetes provider :slight_smile:

You can debug autodiscover events by enabling autodiscover debug flag, use filebeat -e -v -d autodiscover.

Also, we are working on new improvements to ease autodiscover configuration, would you care to explain a bit about your use case and how you plan to use annotations?

Best regards

3

Thank you for the quick reply @exekias. Our use case is that we have a kubernetes cluster where we want filebeat to send logs from all pods by default (which works great), but we want the ability to prevent sending logs from certain pods by setting an annotation. We hoped to use an annotation like logging.mycompany.net/harvest: false in our pods to do this.

Since my previous message, we made some progress by using a simpler annotation name of just harvest: false which seems to work. Here is our filebeat.yml that works:

    filebeat.config:
      prospectors:
        path: ${path.config}/prospectors.d/*.yml
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false
    output.logstash:
      hosts:
        - ${LOGSTASH_HOST}:${LOGSTASH_PORT}
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          in_cluster: true
          tags:
            - "kubernetes"
          include_annotations: ["harvest"]
          templates:
            - condition:
                contains:
                  kubernetes.annotations.harvest: "true"
              config:
                - type: docker
                  containers.ids:
                  - "${data.kubernetes.container.id}"

If we then change the annotation in the pod to harvest.me and change our config to use harvest.me rather than harvest it stops working.

Could the structure of annotations be different somehow in the conditional checks? Thanks very much for the help!

4

Uhm this is interesting, the issue may be related to dots in the name. We may need to add some more debug here to see the autodiscover events. You can run filebeat with -d autodiscover,bus,kubernetes. That should gives a little more info on the annotations included in the event.

Best regards

5

I tried running with -d autodiscover,bus,kubernetes and it does give more information, but only when the condition matches. If the condition does not match, nothing seems to appear in the logs, that is what I was hoping to see. So I can't see the structure of annotations for anything with a . in it.

6

Hi again,

What version of Filebeat are you using? Also, could you paste the log output you see? I think the bus debug selector should give us something here.

Best regards

7

We are on filebeat 6.2.1

Here is the bus output when I deploy a jenkins pod with the harvest.me: false annotation:

2018-02-15T13:09:59.852Z        DEBUG   [bus]   bus/bus.go:55   filebeat: map[config:[0xc4204c8060] start:true host: port:8080 kubernetes:{"annotations":{"harvest.me":"false"},"container":{"id":"","image":"jenkins:latest","name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}} meta:{"kubernetes":{"annotations":{"harvest.me":"false"},"container":{"name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}}}]
2018-02-15T13:09:59.859Z        DEBUG   [bus]   bus/bus.go:55   filebeat: map[config:[0xc4203f6030] stop:true host: kubernetes:{"annotations":{"harvest.me":"false"},"container":{"id":"","image":"jenkins:latest","name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}} meta:{"kubernetes":{"annotations":{"harvest.me":"false"},"container":{"name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}}}]
2018-02-15T13:09:59.860Z        DEBUG   [bus]   bus/bus.go:55   filebeat: map[port:8080 kubernetes:{"annotations":{"harvest.me":"false"},"container":{"id":"","image":"jenkins:latest","name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}} meta:{"kubernetes":{"annotations":{"harvest.me":"false"},"container":{"name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}}} config:[0xc4203e1440] stop:true host:]
2018-02-15T13:09:59.860Z        DEBUG   [bus]   bus/bus.go:55   filebeat: map[host: kubernetes:{"annotations":{"harvest.me":"false"},"container":{"id":"","image":"jenkins:latest","name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}} meta:{"kubernetes":{"annotations":{"harvest.me":"false"},"container":{"name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}}} config:[0xc4201569f0] start:true]
2018-02-15T13:09:59.860Z        DEBUG   [bus]   bus/bus.go:55   filebeat: map[start:true host: port:8080 kubernetes:{"annotations":{"harvest.me":"false"},"container":{"id":"","image":"jenkins:latest","name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}} meta:{"kubernetes":{"annotations":{"harvest.me":"false"},"container":{"name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}}} config:[0xc4201432f0]]
2018-02-15T13:10:02.590Z        DEBUG   [bus]   bus/bus.go:55   filebeat: map[stop:true host:10.105.81.124 kubernetes:{"annotations":{"harvest.me":"false"},"container":{"id":"c879ee6bcf97a6346e685839eea53e43c4bf0e8f8a824e65a65720a87fe9d818","image":"jenkins:latest","name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}} meta:{"kubernetes":{"annotations":{"harvest.me":"false"},"container":{"name":"jenkins"},"labels":{"app":"jenkins","pod-template-hash":"1021400255","serve":"yes","track":"stable"},"namespace":"foundation-dev","node":{"name":"ip-10-105-86-179.eu-west-1.compute.internal"},"pod":{"name":"jenkins-deployment-5465844699-t98d8"}}} config:[0xc420529ec0]]

Here is my autodiscover config, it is slightly different than what I provided previously:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      in_cluster: true
      tags:
        - "kubernetes"
      include_annotations: ["harvest.me"]
      templates:
        - condition:
            and:
              - regexp:
                  kubernetes.namespace: ".*-dev$"
              - not:
                  contains:
                    kubernetes.annotations.harvest.me: "false"
          config:
            - type: docker
              containers.ids:
              - "${data.kubernetes.container.id}"

We are also using a condition on the namespace to only send logs from pods in *-dev namespaces. I don't think it is relevant to our problem, but I wanted to include it just in case.

Again, if I change all instances of harvest.me to harvest the logic works as expected and no logs are sent.

Thanks again!

8

Yes,

The issue is with the dot, we are not converting {"annotations":{"harvest.me":"false"}} to {"annotations":{"harvest": {"me":"false"}}}.

Recently we merged this PR: https://github.com/elastic/beats/pull/6203 to convert all dotted annotations. It will move harvest.me to harvest_me, and you won't have the problem.

As it is a broken change we are not planning to backport it to 6.2.X, but release it with 6.3.0.

9

Thanks for the help @exekias we will keep an eye out for 6.3.

10

This topic was automatically closed after 21 days. New replies are no longer allowed.