Filebeat Fails to Process Symlinked Log Files Despite Symlink Configuration Enabled

Elastic Stack Beats
1

I am encountering an issue with Filebeat where it fails to process symlinked log files, even though I have configured the symlink handling to be enabled. Below are the details of my configuration and the error messages I’m receiving.

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: filestream
        id: kubernetes-container-logs-${data.kubernetes.node.name}-${data.kubernetes.pod.name}-${data.kubernetes.container.id}-${data.kubernetes.pod.uid}
        paths:
          - /var/log/containers/*-${data.kubernetes.container.id}.log
        parsers:
          - container: ~
        prospector:
        scanner:
          fingerprint.enabled: false
          symlinks: true
        file_identity.fingerprint: ~
        clean_removed: true

I have checked the log files in /var/log/containers and confirmed that they are symlinks. However, I still receive the following error message:

{"log.level":"debug","@timestamp":"2024-09-26T07:31:38.571Z","log.logger":"scanner","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).GetFiles","file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create an ingest target for file \"/var/log/containers/log-gen-6cfd5d4cfc-gkbmp_monitoring_log-gen-80a3482ec2656007c1308be932a56dc3420d901cce4bbf5336e0ee184dc88d3d.log\": file \"/var/log/containers/log-gen-6cfd5d4cfc-gkbmp_monitoring_log-gen-80a3482ec2656007c1308be932a56dc3420d901cce4bbf5336e0ee184dc88d3d.log\" is a symlink and they're disabled","service.name":"filebeat","ecs.version":"1.6.0"}

Additional Information:

• I have ensured that the symlink handling is enabled in my Filebeat configuration.
• I have deleted the meta.json and log.json files to see if it resolves the issue, but the problem persists.
• I have verified the permissions and existence of the target files for the symlinks.

I appreciate your help!

2

Set to setup.template.enabled: false because we cannot use same template

# logging.level: debug

filebeat.inputs:
  - type: filestream
    id: auditbeat
    fields:
      auditbeat: true
    paths:
      - /var/log/auditbeat/*
    parsers:
      - ndjson:
          target: ""
          overwrite_keys: true

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        id: kubernetes-container-logs-${data.kubernetes.node.name}-${data.kubernetes.pod.name}-${data.kubernetes.container.id}-${data.kubernetes.pod.uid}
        paths:
          - /var/log/containers/*-${data.kubernetes.container.id}.log

output.elasticsearch:
  logging.level: debug
  logging.selectors: ["*"]
  hosts: ["${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}"]
  compression_level: 9
  indices:
    - index: "auditbeat-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.equals:
        fields.auditbeat: true
    - index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.not.equals:
        fields.auditbeat: true

# For debug purpose
# output.console:
#   enabled: true
#   codec.json:
#     # Pretty-print JSON event
#     pretty: false

setup.template.enabled: false
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
setup.ilm.enabled: true
setup.ilm.mode: auto