# Filebeat autodiscover exclude\_lines regex

**URL:** <https://discuss.elastic.co/t/filebeat-autodiscover-exclude-lines-regex/166559>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 31, 2019, 1:14pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-exclude-lines-regex/166559 "2019-01-31T13:14:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![danijelh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danijelh/32/40419_2.png) [@danijelh](https://discuss.elastic.co/u/danijelh)\
**Post date:** [January 31, 2019, 1:14pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-exclude-lines-regex/166559/1 "2019-01-31T13:14:51Z")

</div>

Hi.

I am trying to exclude certain lines from pushing them to the ELK stack. We are using dockers and everything is working fine, but excluded lines are still being pushed to the ELK.

Our config:

```
filebeat.registry_file: /var/log/containers/filebeat_registry

queue.mem:
  events: 2048
  flush.min_events: 512
  flush.timeout: 5s

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
# Autodiscover docker containers and parse logs
  providers:
    - type: docker
      templates:
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ['\d{2}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}\sINFO.*metrics-middleware']

filebeat.inputs:
  - type: docker
    containers.ids:
      - "*"
    processors:
      - add_docker_metadata: ~
    fields:
      microservice: true
      environment: ${FILEBEAT_ENV}
    fields_under_root: true

```

Example of logs:

```
2019-01-31 11:32:49.289 INFO 24 --- [node] metrics-middleware : sending system metrics
2019-01-31 11:32:54.289 INFO 24 --- [node] metrics-middleware : sending system metrics
2019-01-31 11:32:59.289 INFO 24 --- [node] metrics-middleware : sending system metrics
2019-01-31 11:33:04.289 INFO 24 --- [node] metrics-middleware : sending system metrics

```

These line are still being pushed to the ELK. I have tested my regex on go playground and it is working fine.

[https://play.golang.org/p/V8p-0JHXc1i](https://play.golang.org/p/V8p-0JHXc1i)

---

<div class="post-metadata">

**Author:** ![danijelh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danijelh/32/40419_2.png) [@danijelh](https://discuss.elastic.co/u/danijelh)\
**Post date:** [January 31, 2019, 3:02pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-exclude-lines-regex/166559/2 "2019-01-31T15:02:24Z")

</div>

I have added these lines under the **filebeat.inputs** :

```
json.keys_under_root: true
json.add_error_key: true
json.message_key: log
exclude_lines: ['\d{2}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}\sINFO.*metrics-middleware']

```

Now it looks like those messages are not being sent to the ELK stack anymore. I wonder what does exclude\_lines under filebeat.autodiscover do?

---

<div class="post-metadata">

**Author:** ![danijelh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danijelh/32/40419_2.png) [@danijelh](https://discuss.elastic.co/u/danijelh)\
**Post date:** [January 31, 2019, 4:39pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-exclude-lines-regex/166559/3 "2019-01-31T16:39:17Z")

</div>

Hm. I am getting now errors in ELK such as

`Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}`

my logs look like:

```
{"log":"2019-01-31 14:08:59.284 INFO 24 --- [node] metrics-middleware : sending request metrics\n","stream":"stdout","time":"2019-01-31T14:08:59.28447716Z"}
{"log":"2019-01-31 14:08:59.285 INFO 24 --- [node] metrics-middleware : sending request metrics\n","stream":"stdout","time":"2019-01-31T14:08:59.285860708Z"}
{"log":"2019-01-31 14:08:59.288 INFO 24 --- [node] metrics-middleware : sending system metrics\n","stream":"stdout","time":"2019-01-31T14:08:59.288782568Z"}
{"log":"2019-01-31 14:08:59.290 INFO 24 --- [node] metrics-middleware : sending system metrics\n","stream":"stdout","time":"2019-01-31T14:08:59.290303786Z"}
```

---

<div class="post-metadata">

**Author:** ![danijelh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danijelh/32/40419_2.png) [@danijelh](https://discuss.elastic.co/u/danijelh)\
**Post date:** [January 31, 2019, 5:02pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-exclude-lines-regex/166559/4 "2019-01-31T17:02:01Z")

</div>

I made it now as:

```
filebeat.inputs:
  - type: docker
    containers.ids:
      - "*"
    processors:
      - add_docker_metadata: ~
    fields:
      microservice: true
      environment: '${FILEBEAT_ENV}'
    fields_under_root: true
    exclude_lines: ['\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}\s+INFO.*metrics.*']

```

currently there are no unwanted lines in the ELK, let's see if everything else is coming in.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2019, 5:02pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-exclude-lines-regex/166559/5 "2019-02-28T17:02:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.