Filebeat query EKS worker node /var/log

Hi,

I'm trying to ship EKS worker node auth.log, syslog and audit.log files which located under /var/log.
I've deploy filebeat and logstash in EKS cluster however I saw under filebeat pods there's a lot of error log stated

{"log.level":"error","@timestamp":"2023-09-11T14:01:23.876Z","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/matchers.go","file.line":155},"message":"Error extracting container id - source value contains matcher's logs_path, however it is too short to contain a Docker container ID.","service.name":"filebeat","ecs.version":"1.6.0"}

This is my filebeat config

- type: log
  paths:
    - '/var/log/auth.log'
  processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      matchers:
      - logs_path:
          logs_path: "/var/log/"
  - add_tags:
      tags: [authlog]
      target: "log_category"

Hi @xUmaRix,

Welcome back! Have you checked the health of your Kubernetes pod as recommended in the troubleshooting guide?

Can you elaborate on why you are using the add_kubernetes_metadata processor when collecting system-level logs? The add_kubernetes_metadata processor annotates each event with relevant metadata based on the Kubernetes pod from which the event originated. It seems you are collecting logs at a different level (system) in the hierarchy but want to enrich with pod level metadata.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.