I am using filebeat(v 8.1.0) to pull logs from microservices deployed in kubernetes env. Right now, there is a huge amount of meta data generated by filebeat apart from the actual log message. Is there a way cut it sort to only log message and the source pod/app?
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
templates:
- condition:
contains:
kubernetes.container.name: "my-api"
config:
- type: container
paths:
- "/var/log/containers/*-${data.kubernetes.container.id}.log"
json.keys_under_root: true
json.message_key: message
processors:
- drop_event:
when:
contains:
message: "/health_check HTTP/"
- drop_fields:
fields: ["kubernetes.labels.pod-template-hash", "kubernetes.replicaset.name", "kubernetes.node", "log","ecs", "agent.type", "agent.hostname", "agent.ephemeral_id", "agent.id", "kubernetes.namespace_uid", "kubernetes.namespace_labels"]