# Filebeat multiline filter with autodiscover

**URL:** <https://discuss.elastic.co/t/filebeat-multiline-filter-with-autodiscover/217432>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 31, 2020, 5:05pm UTC](https://discuss.elastic.co/t/filebeat-multiline-filter-with-autodiscover/217432 "2020-01-31T17:05:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ztasre](https://avatars.discourse-cdn.com/v4/letter/z/ecb155/32.png) [@ztasre](https://discuss.elastic.co/u/ztasre)\
**Post date:** [January 31, 2020, 5:05pm UTC](https://discuss.elastic.co/t/filebeat-multiline-filter-with-autodiscover/217432/1 "2020-01-31T17:05:01Z")

</div>

Filebeat version: 7.5.2

I am trying to get my application running on Kubernetes with the ELK stack to do logging. The application is written in Java so I need to be able to able to ingest multiline stack traces as a single log message, I have a regex in Filebeat that does this. I want to apply this multiline filtering only to pods with the Kubernetes app label "my-app", the logs for everything else should be ingested regularly.

I am having a lot of trouble setting up autodiscovery, my config (from my k8s filebeat manifest) is below:

```auto
filebeat.yml: |-
    filebeat.autodiscover:
      providers:
          - type: kubernetes
            templates:
                - condition:
                    equals:
                        kubernetes.labels.app: "my-app"
                  config:
                      - type: container
                        #containers.ids:
                        # - "${data.docker.container.id}"
                        # - "${data.kubernetes.container.id}" #"*"
                        paths:
                          - /var/log/container/*-${data.kubernetes.container.id}.log
                        multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
                        multiline.negate: true
              filebeat.config:
     #inputs: # prospectors formerly
        # Mounted `filebeat-prospectors` configmap, keep it in the prospector dir for now TODO
        #path: ${path.config}/prospectors.d/*.yml
        # Reload prospectors configs as they change:
        #reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        # all modules are disabled currently
        reload.enabled: false
    output.logstash:
      hosts: ['logstash-service:5044']
          multiline.match: after

```

I tried adding a paths field and I tried using containers.ids. I still can't see my logs in Kibana using the above config. When using "containers.ids" Filebeat was complaining that a "paths" needs to be set, I set it and it still doesn't work. My cluster is working correctly since using the config below works but ignores other non-"my-app" logs that I want:

```auto
kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true

```

If autodiscovery is not the way to do this then what is? I want a multiline filter applied to my Spring app logs (so stack traces get grouped as one message) and I want default (no filtering) applied to all other logs. Potentially I may want to add more custom rules in the future too. I have like 15 or so services running already ideally I don't want to add a rule for all of them, only the ones for which I don't want the default behavior for logging.

---

<div class="post-metadata">

**Author:** ![ztasre](https://avatars.discourse-cdn.com/v4/letter/z/ecb155/32.png) [@ztasre](https://discuss.elastic.co/u/ztasre)\
**Post date:** [February 4, 2020, 3:49pm UTC](https://discuss.elastic.co/t/filebeat-multiline-filter-with-autodiscover/217432/2 "2020-02-04T15:49:00Z")

</div>

I got it to work using hints based autodiscover. If someone has a similar issue look at the docs here:  
[https://www.elastic.co/guide/en/beats/filebeat/master/configuration-autodiscover-hints.html](https://www.elastic.co/guide/en/beats/filebeat/master/configuration-autodiscover-hints.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2020, 3:49pm UTC](https://discuss.elastic.co/t/filebeat-multiline-filter-with-autodiscover/217432/3 "2020-03-03T15:49:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.