Filebeat 8.4.1: Autodiscovery and processor not working as expected as per the doc

Elastic Stack Beats
,
1

Hi.
I am new to filebeat (previously using fluentbit to send kubernetes logs to graylog) and I am struggling with some things that seems not to work as it is in the doc. Maybe I am missing something, so please be benevolent with me.
I have spent two days on this, that's why I am so verbous and specific, sorry for the length.

With the filebeat config and Kubernetes pod annotations below I am getting the traefik2 and filebit itself logs into my graylog instance.
BUT

  1. When I replace target: "meta" with target: "" in the decode_json_fields processor (or completely remove it) the logs do not go to the output (verified with output.console). It just stops sending json logs. No clue nor error in the logs.
    Anyone knows why or how to workaround it?

  2. In the pod example below, if I enable any of the annotations in the pod aside the co.elastic.logs/enabled:"true", the logs also are discarded/not sent to output (also checked with output.console).
    I am doing something wrong?

  3. I need to add a field to the log with the value of the pod annotation app.kubernetes.io/instance
    For example: app.kubernetes.io/instance=traefik2-back
    But any attempt of using annotations has been unsuccessful.
    Any help will be wonderful.

  4. As far as I understand processors specified in the config file are applied to any log comming through the autodiscover. I tried to apply specific processors to different containers using annotations, but they didn't work either.
    There is any other way to choose which processor should be used for each log ?

Thanks a lot if you read it all :slight_smile:
If you can also help will be great! :wink:

   # File: filebeat.yaml
   # Image: docker.elastic.co/beats/filebeat:8.4.1
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config.enabled: false
          #hints.default_config:
          #  type: container
          #  paths:
          #    - /var/log/containers/*${data.kubernetes.container.id}.log
          #  tail_files: true
    processors:
      - decode_json_fields:
          fields: ["message"] 
          process_array: false
          max_depth: 1
          target: "meta"  
          overwrite_keys: true
          add_error_key: true

    output.logstash:
      hosts: ["gelf-inputs.graylog.svc.cluster.local:5044"]

    # File: deployment.yaml
     [...]
     deployment:
       podAnnotations:
          co.elastic.logs/enabled: "true"
          #co.elastic.logs.traefik2-back/enabled: "true"
          #co.elastic.logs.traefik2-back/json.keys_under_root: "true"
          #co.elastic.logs.traefik2-back/json.overwrite_keys: "false"
          #co.elastic.logs.traefik2-back/json.expand_keys: "true"
          #co.elastic.logs.traefik2-back/close_renamed: "true"
          #co.elastic.logs.traefik2-back/close_removed: "true"
     [...]

Hope you can thow some light on my doubts.
Thanks again

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.