Hi
I am using "add_docker_metadata" processor.
And when I looked at events published to ES, I see most of the docker metadata common or duplicate under "_source" as well as "docker" fields.
Is there a way to eliminate one of these sets. They seem redundant.
"_source" : {
"container" : {
"image" : {
"name" : "sha256:922c269cf957ec17d66cb5acdfbdd45d420568ec8a6ccd23c6e62b7a7aacb7c8"
},
"name" : "k8s_xxxwin0_darwin0-6b4f97655-fmdm4_default_67d93d84-f5bb-11e9-bf32-00505690ff20_1",
"id" : "7f9b94c2b590c04ba123bc487a9a5893060484c9bead76db3c57fca3dde5a083",
"labels" : {
"annotation_io_kubernetes_pod_terminationGracePeriod" : "30",
...
"docker" : {
"container" : {
"labels" : {
"annotation_io_kubernetes_pod_terminationGracePeriod" : "30",
"io_kubernetes_container_logpath" : "/var/log/pods/67d93d84-f5bb-11e9-bf32-00505690ff20/xxxwin0/1.log",
....
Appreciate any help.
Thx
B.M
(Badre)
October 28, 2019, 1:31pm
2
Hi John,
Could you share your config file?
Hi Badre
Thanks for your reply. Here is most of my filebeat config file.
( Feel free to suggest other improvements if you see some. I am still new to filebeat )
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: default
labels:
k8s-app: xxx
app: xxx
data:
filebeat.yml: |-
processors:
- add_cloud_metadata:
- add_host_metadata:
- add_docker_metadata:
labels.dedot: true
annotations.dedot: true
- drop_fields:
#fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "host.name", "ecs.version", "input.type"]
fields:
- "agent.ephemeral_id"
- "agent.hostname"
- "agent.id"
- "agent.type"
- "agent.version"
- "host.name"
- "ecs.version"
- "input.type"
ignore_missing: true
filebeat.autodiscover:
providers:
- type: docker
# this input section is for module eg: mongo/nginx specific
templates:
- condition:
contains:
docker.container.name: echo
config:
- module: nginx
enabled: true
access:
input:
type: container
stream: "stdout"
containers.ids:
- ${data.docker.container.id}
error:
input:
type: container
stream: "stdout"
containers.ids:
- ${data.docker.container.id}
- condition:
contains:
docker.container.name: darwin
config:
- module: mongodb
enabled: true
log:
input:
type: docker
containers.ids:
- ${data.docker.container.id}
filebeat.inputs:
- type: log
# this input section is for system log files
enabled: true
paths:
- /var/log/*.log
- /var/log/messages
- type: container
# this input section is for general application docker/containers logs
enabled: true
stream: all
paths:
- /var/lib/docker/containers/*/*.log
include_lines: ['"pod":']
json.keys_under_root: true
json.ignore_decoding_error: true
json.add_error_key: true
json.overwrite_keys: true
json.message_key: log
# Avoid parsing exceptions due to name conflicts
processors:
- rename:
fields:
- from: "service"
to: "service-name"
- from: "error"
to: "service-error"
- from: "url"
to: "service-url"
ignore_missing: true
fail_on_error: false
B.M
(Badre)
October 28, 2019, 4:32pm
4
Hi @john_eapen i am sorry but i meant to comment on another post. But i think the sitting you are looking for is drop_fields processor , with this setting your can drop other fields as-well except "type" and "@timestamp " since these are default fields to filebeat.
I hope this answers your question
system
(system)
Closed
November 25, 2019, 4:39pm
5
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.