Removing unwanted data in Filebeats / Logstash

Elastic Stack Logstash
1

I started ingesting Kubernetes logs into ElasticSearch, using the Filebeat Input Container:
Filebeat -> Logstash -> ElasticSearch

My filebeat.yml looks like this:

- type: container
  paths:
    - /var/lib/docker/containers/*/*.log
  fields:
    logs_type: "k8s"
  fields_under_root: true
  multiline.type: pattern
  multiline.pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
  multiline.negate: false
  multiline.match: after

processors:
- add_docker_metadata:
    host: "unix:///var/run/docker.sock"

output:
  logstash:
    enabled: True
    hosts:
      - logstash1:5043
      - logstash2:5043
    loadbalance: true

  1. Is there a way to drop filed names that I don't believe I'll use, .e.g: container.labels.org_label-schema_*

  2. What is the best way to drop messages from an entire container (e.g. no logs for etcd container)?

  3. What is the best way to drop a specific message, form a specific container? For example, my solr container prints this useless message every second:

2021-01-01 00:00:00.000 INFO  (qtp1107530534-20) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={} status=0 QTime=6
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.