Filebeat rename processor with conditions

Elastic Stack Beats
1

Hi! I'm trying to rename some fields from kubernetes annotations based on an when conditions, due to not finding any good resources, I was wondering if someone of you could help me with this.

My goal is to rename events which contain kubernetes.annotations.myapp/sidecar = 'true' and when kubernetes.annotations.myapp/sidecar.type = kubernetes.container.name. If none of these conditions go through, I want to rename as a default (in the other when).

Is this possible? I did not find any good examples on how to do, if possible more than one condition under one processor (global).

- rename:
          when:
              equals: kubernetes.annotations.myapp/sidecar.source: kubernetes.container.type 
                fields:
                  - from: "kubernetes.labels.myapp-service"
                    to: "_service"
                  - from: "kubernetes.labels.myapp-token"
                    to: "_token"
                  - from: "kubernetes.annotations.myapp/sidecar.type"
                    to: "type"
          when:
            or:
              - not.has_fields: ['kubernetes.annotations.myapp/sidecar']
              - not.equals:
                  kubernetes.annotations.myapp/sidecar: 'true'
                 - from: "kubernetes.labels.myapp-service"
                    to: "_service"
                  - from: "kubernetes.labels.myapp-token"
                    to: "_token"
                  - from: "kubernetes.labels.myapp-type"
                    to: "type"
2

I guess you are using auto-discovery? Instead of global filter per event, you might consider to use hints or templates to configure the required processor on container startup and save some CPU.

The syntax of processors in filebeat goes like this:

processors:
- <name>:
    <processor settings>
    when:
      <conditional>

That is, you have to switch order.

Right now you can not compare 2 event fields using equal. The right hand side of equals must be string/numeric/boolean constant. If kubernetes.container.type is an event value, then the condition is not supported (feel free to open a feature request).

Anyways, more correct solution is:

processors:
- rename:
    fields:
      - from: "kubernetes.labels.myapp-service"
        to: "_service"
      - from: "kubernetes.labels.myapp-token"
        to: "_token"
      - from: "kubernetes.annotations.myapp/sidecar.type"
        to: "type"
    when:
      equals:
        'kubernetes.annotations.myapp/sidecar.source': ...   # <- must be some constant value

- rename:
    fields:
      - from: "kubernetes.labels.myapp-service"
        to: "_service"
      - from: "kubernetes.labels.myapp-token"
        to: "_token"
      - from: "kubernetes.labels.myapp-type"
        to: "type"
    when:
      or:
        - not.has_fields: ['kubernetes.annotations.myapp/sidecar']
        - not.equals:
          kubernetes.annotations.myapp/sidecar: true
3

Thanks alot for your feedback. Yes I am using autodiscover in Kubernetes (should have included that.. ), I should also mention that this is steps that we are doing in order to get it functional with a logstash pipeline down the road (k8s->filebeat->logstash->elasticsearch). This scenario is meant for sidecar containers. I know the parsing itself could be done through hints/annotations, my issue is more related to extracting and performing logic based upon annotations. In my mind every pod which has more than one running container would specify sidecar configurations so that down the line it would work with a logstash pipeline dependent on the type(thats why i rename/extract it) being added to the log event.

As for kubernetes.container.type I meant kubernetes.container.name, I think that should be added to every event.

Thanks.

4

Hi @havlan,

As far as I know autodiscover should be reporting the container name, please open a new thread if that's not the case for you

5

It does (it was just a typo from me), I was just wondering whether the if/else approach would work. I'll try @steffens approach and report back if I have any issues.

Thank you both for your quick answers.

6

You can not compare an event field with another event field. That is, this condition is not supported:

equals:
  'kubernetes.annotations.myapp/sidecar.source': 'kubernetes.container.name'

You will have to find some other method (constant value/flag) to filter upon.

7

I see. Thanks for the clarification. It's hard to inject a value or something into a container from a pod manifest, but I'll look more into it.
This sidecar use case with annotation logic might not be possible then.

8

Github feature request/issue open https://github.com/elastic/beats/issues/8764

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.