Greetings,
We have filebeat deployed to a kubernetes cluster (our QA instance).
Filebeat is picking up logs for the containers and sending them to our logstash server, which ends up in our elasticsearch. However, the events are not getting the correct kubernetes metadata added. Our setup has worked well for a couple of years, but after a restart of filebeat , logging is not coming in as expected to our logstash.
My team's can no longer search for their events in kibana, due to messages being associated with a different kubernetes pod, container, deployment, etc.
Scenario that occurs for my pods...
In a published event, the wrong kubernetes data was added . Instead of getting the kuberentes data for my running pod: or host: dsp-spring-boot-pr-1538-generic-http-server-78c4f9757-mq7kt , It added the metadata from the pod or host: snoop-doghouse-pr-4989-generic-http-server-76d9dfc678-6swzg, which is another pod on the same kubernetes node, but logs to a different file.
I've tried multiple config adjustments, but here is what I have and used to work fine:
filebeat.yml
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
hints.default_config:
type: container
fields:
env: "qa"
paths:
- /var/log/containers/*.log
multiline.pattern: '^[[:space:]]'
multiline.match: after
multiline.negate: false
ignore_older: 3h # ignore files that were modified 3 hours ago
clean_inactive: 4h # removes the state of a file after 4h (it must be > ignore_older + scan_frequency)
close_inactive: 1m # close file handle after 1 minute of inactivity (file can be reopened after scan_frequency)
scan_frequency: 1m # check for new files every 1 minute
harvester_limit: 50 # limit the number of harvesters to 50 at a time
filebeat.config:
modules:
path: /etc/filebeat/modules.d/*.yml
reload.enabled: false
inputs:
enabled: true
path: /etc/filebeat/*.yml
reload.enabled: true
reload.period: 10s
logging.level: debug
processors:
- add_cloud_metadata: null
- add_host_metadata:
netinfo.enabled: true
- add_kubernetes_metadata:
host: $${NODE_NAME}
in_cluster: true
labels.dedot: true # https://github.com/elastic/beats/issues/8773
annotations.dedot: true
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- drop_event:
when:
equals:
kubernetes.container.name: filebeat
- drop_event:
when:
and:
- equals:
kubernetes.namespace: jenkins
- equals:
fields.env: qa
http.enabled: true
http.port: 5066
output:
elasticsearch:
hosts:
- http://elasticsearch-master:9200
enabled: false
file:
enabled: false
logstash:
enabled: true
hosts: "logstash-0.mkeymgmt.com:5044"
ssl:
certificate_authorities:
- "/tmp/filebeat/root-ca.pem"
certificate: "/tmp/filebeat/client.pem"
key: "/tmp/filebeat/client.key"
verification_mode: none
supported_protocols:
- TLSv1.2
output.file:
filename: filebeat
number_of_files: 5
path: /usr/share/filebeat/data
rotate_every_kb: 10000
Log event from the pod:
{"log":"{\"@timestamp\":\"2024-02-29T21:07:47.453Z\",\"severity\":\"INFO\",\"application\":\"DSP\",\"trace\":\"\",\"span\":\"\",\"correlation_id\":\"\",\"pid\":\"7\",\"thread\":\"http-nio-8080-exec-8\",\"class\":\"c.l.dsp.utils.MetricsLogInterceptor\",\"message\":\"Endpoint=http://localhost:35481/v1/offerings, Execution-time-ms=7, Remote-ip=127.0.0.1, Forwarded-For=, Response-Code=200\",\"x_forwarded_for\":\"\",\"incap_client_ip\":\"\",\"host\":\"dsp-spring-boot-pr-1538-generic-http-server-78c4f9757-mq7kt\"}\n","stream":"stdout","time":"2024-02-29T21:07:47.453424719Z"}
however, I could not find this event in my dashbaord based on what I usually do (I search for records by pod name)
When looking at the published event in logstash, there is a mismatch between from which log file the event came, and the kubernetes info reflected.
example, I see my event is correct, but from this excerpt, I am showing the data came in for a different pod
"kubernetes": {
"deployment": {
"name": "snoop-doghouse-pr-4989-generic-http-server"
},
"namespace_uid": "94c1c379-7bc2-4937-951a-7bfb41ce84f8",
"namespace": "greenhouse-pr-4989",
"namespace_labels": {
"pullRequest": "4989",
"gitBranch": "SNAP-2890",
"deleteStrategy": "branch-delete",
"repoName": "greenhouse",
"kubernetes_io/metadata_name": "doghouse-pr-4989"
},
"labels": {
"release": "snoop-doghouse-pr-4989",
"app": "generic-http-server",
"gitSha": "a0200de9ec1e2d99630d968337a28e5c810db663",
"pod-template-hash": "76d9dfc678"
},
Any guidance would be greatly appreciated!
Thanks