Filebeat multiline filter with autodiscover

Filebeat version: 7.5.2

I am trying to get my application running on Kubernetes with the ELK stack to do logging. The application is written in Java so I need to be able to able to ingest multiline stack traces as a single log message, I have a regex in Filebeat that does this. I want to apply this multiline filtering only to pods with the Kubernetes app label "my-app", the logs for everything else should be ingested regularly.

I am having a lot of trouble setting up autodiscovery, my config (from my k8s filebeat manifest) is below:

filebeat.yml: |-
          - type: kubernetes
                - condition:
                      - type: container
                        #  - "${}"
                        #    -  "${}" #"*"
                          - /var/log/container/*-${}.log
                        multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
                        multiline.negate: true
     #inputs: # prospectors formerly
        # Mounted `filebeat-prospectors` configmap, keep it in the prospector dir for now TODO
        #path: ${path.config}/prospectors.d/*.yml
        # Reload prospectors configs as they change:
        #reload.enabled: false
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        # all modules are disabled currently
        reload.enabled: false
      hosts: ['logstash-service:5044']
          multiline.match: after

I tried adding a paths field and I tried using containers.ids. I still can't see my logs in Kibana using the above config. When using "containers.ids" Filebeat was complaining that a "paths" needs to be set, I set it and it still doesn't work. My cluster is working correctly since using the config below works but ignores other non-"my-app" logs that I want:

kubernetes.yml: |-
    - type: docker
      - "*"
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
        - add_kubernetes_metadata:
            in_cluster: true

If autodiscovery is not the way to do this then what is? I want a multiline filter applied to my Spring app logs (so stack traces get grouped as one message) and I want default (no filtering) applied to all other logs. Potentially I may want to add more custom rules in the future too. I have like 15 or so services running already ideally I don't want to add a rule for all of them, only the ones for which I don't want the default behavior for logging.

I got it to work using hints based autodiscover. If someone has a similar issue look at the docs here:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.