Filebeat autodiscover exclude_lines regex

Elastic Stack Beats
1

Hi.

I am trying to exclude certain lines from pushing them to the ELK stack. We are using dockers and everything is working fine, but excluded lines are still being pushed to the ELK.

Our config:

filebeat.registry_file: /var/log/containers/filebeat_registry

queue.mem:
  events: 2048
  flush.min_events: 512
  flush.timeout: 5s

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
# Autodiscover docker containers and parse logs
  providers:
    - type: docker
      templates:
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ['\d{2}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}\sINFO.*metrics-middleware']

filebeat.inputs:
  - type: docker
    containers.ids:
      - "*"
    processors:
      - add_docker_metadata: ~
    fields:
      microservice: true
      environment: ${FILEBEAT_ENV}
    fields_under_root: true

Example of logs:

2019-01-31 11:32:49.289 INFO 24 --- [node] metrics-middleware : sending system metrics
2019-01-31 11:32:54.289 INFO 24 --- [node] metrics-middleware : sending system metrics
2019-01-31 11:32:59.289 INFO 24 --- [node] metrics-middleware : sending system metrics
2019-01-31 11:33:04.289 INFO 24 --- [node] metrics-middleware : sending system metrics

These line are still being pushed to the ELK. I have tested my regex on go playground and it is working fine.

https://play.golang.org/p/V8p-0JHXc1i

2

I have added these lines under the filebeat.inputs:

json.keys_under_root: true
json.add_error_key: true
json.message_key: log
exclude_lines: ['\d{2}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}\sINFO.*metrics-middleware']

Now it looks like those messages are not being sent to the ELK stack anymore. I wonder what does exclude_lines under filebeat.autodiscover do?

3

Hm. I am getting now errors in ELK such as

Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

my logs look like:

{"log":"2019-01-31 14:08:59.284 INFO 24 --- [node] metrics-middleware : sending request metrics\n","stream":"stdout","time":"2019-01-31T14:08:59.28447716Z"}
{"log":"2019-01-31 14:08:59.285 INFO 24 --- [node] metrics-middleware : sending request metrics\n","stream":"stdout","time":"2019-01-31T14:08:59.285860708Z"}
{"log":"2019-01-31 14:08:59.288 INFO 24 --- [node] metrics-middleware : sending system metrics\n","stream":"stdout","time":"2019-01-31T14:08:59.288782568Z"}
{"log":"2019-01-31 14:08:59.290 INFO 24 --- [node] metrics-middleware : sending system metrics\n","stream":"stdout","time":"2019-01-31T14:08:59.290303786Z"}
4

I made it now as:

filebeat.inputs:
  - type: docker
    containers.ids:
      - "*"
    processors:
      - add_docker_metadata: ~
    fields:
      microservice: true
      environment: '${FILEBEAT_ENV}'
    fields_under_root: true
    exclude_lines: ['\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}\s+INFO.*metrics.*']

currently there are no unwanted lines in the ELK, let's see if everything else is coming in.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.