Parse json from selected pods only

[igniti0n (Kc)]

Hello!
I'm having an issue getting any parsing rules to work within my filebeat deployments. We have multiple services that need separate parsing rules that end up in the "message" field. Ideally i would like to be able to parse all logs that don't fall within certain templates to have the generic rules applied and the ones i specify to have the extra rules added. I'm pretty sure my issue here is syntax and I'm not too sure the right order to get this all working. Any help would be greatly appreciated. Using filebeat 7.5.0

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    #filebeat.inputs:
    #- type: container
    #  paths:
    #    - /var/log/containers/*.log
    #  processors:
    #    - add_kubernetes_metadata:
    #        host: ${NODE_NAME}
    #        matchers:
    #        - logs_path:
    #            logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
      templates:
        - condition:
              equals:
                kubernetes.annotations.json_logs: "true"
                config:
                  processors:
                      - decode_json_fields:
                          fields: ["message"]
                          target: "json_message"
                          process_array: true
          processors:
            - add_cloud_metadata:
            - add_host_metadata:

    cloud.id: 
    cloud.auth:

[jsoriano (Jaime Soriano)]

Hey @igniti0n, welcome to discuss!

I see you are trying to combine hints-based autodiscover, and autodiscover templates. Even if it is possible to combine them, they use to lead to unexpected results, because they are both applied. In your case it would be adding the configuration of the default_config and the template as two separate configurations for the same pods.

So I would recommend to use only one kind of configuration, and in your case, as you want to collect logs from all pods, I would suggest you to use hints-based autodiscover, that applies some configuration to all pods by default.

You could try to add the decode_json_fields processor to your default input config, and remove the templates, something like this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            processors:
              - decode_json_fields:
                  when:
                    kubernetes.annotations.json_logs: "true"
                  fields: ["message"]
                  target: "json_message"
                  process_array: true