Filebeat in k8s pod cannot read log file : No such file or directory

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      fields_under_root: true
      fields:
        evn: lcdp-dev
        log-type: lcdp-app
        server: lcdp
        language: java
      tail_files: true
      paths:
        - /var/log/containers/*_apaas_xjsd*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - drop_fields:
            fields:
              - host
              - ecs
              - stream
            ignore_missing: true

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      node: ${NODE_NAME}
    #      hints.enabled: true
    #      hints.default_config:
    #        type: container
    #        paths:
    #          - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    output.logstash:
      hosts: ['${LOGSTASH_HOST:logstash}:${LOGSTASH_PORT:514}']
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.17.8
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: LOGSTASH_HOST
          value: "10.20.13.35"
        - name: LOGSTASH_PORT
          value: "5151"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /data/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate



filebeat pod log

2022-12-28T06:06:31.952Z        INFO    instance/beat.go:697    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs] Hostfs Path: [/]
2022-12-28T06:06:31.952Z        INFO    instance/beat.go:705    Beat ID: a188c828-2598-40ed-b376-ad1db88d8927
2022-12-28T06:06:34.953Z        WARN    [add_cloud_metadata]    add_cloud_metadata/provider_aws_ec2.go:79       read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.
2022-12-28T06:06:34.958Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2022-12-28T06:06:34.958Z        INFO    [beat]  instance/beat.go:1051   Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "a188c828-2598-40ed-b376-ad1db88d8927"}}}
2022-12-28T06:06:34.958Z        INFO    [beat]  instance/beat.go:1060   Build info      {"system_info": {"build": {"commit": "692b4aac606e457bd2f5ef092d2d23c2fa950828", "libbeat": "7.17.8", "time": "2022-12-03T00:31:52.000Z", "version": "7.17.8"}}}
2022-12-28T06:06:34.958Z        INFO    [beat]  instance/beat.go:1063   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":16,"version":"go1.18.5"}}}
2022-12-28T06:06:34.961Z        INFO    [beat]  instance/beat.go:1067   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-08-30T05:57:35Z","containerized":true,"name":"l-sj-lcdp-work-dev01","ip":["127.0.0.1/8","::1/128","10.20.13.13/24","fe80::250:56ff:fea7:3c32/64","172.17.0.1/16","fe80::42:9bff:fe36:87c2/64","10.42.1.0/32","fe80::8dd:9cff:fe71:f59b/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64"],"kernel_version":"3.10.0-1160.71.1.el7.x86_64","mac":["00:50:56:a7:3c:32","02:42:9b:36:87:c2","0a:dd:9c:71:f5:9b","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0}}}
2022-12-28T06:06:34.961Z        INFO    [beat]  instance/beat.go:1096   Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 8, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-12-28T06:06:31.530Z"}}}
2022-12-28T06:06:34.961Z        INFO    instance/beat.go:291    Setup Beat: filebeat; Version: 7.17.8
2022-12-28T06:06:34.962Z        INFO    [publisher]     pipeline/module.go:113  Beat name: l-sj-lcdp-work-dev01
2022-12-28T06:06:34.963Z        WARN    beater/filebeat.go:202  Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-12-28T06:06:34.963Z        INFO    [monitoring]    log/log.go:142  Starting metrics logging every 30s
2022-12-28T06:06:34.963Z        INFO    instance/beat.go:456    filebeat start running.
2022-12-28T06:06:34.964Z        INFO    memlog/store.go:119     Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2022-12-28T06:06:34.964Z        INFO    memlog/store.go:124     Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2022-12-28T06:06:34.964Z        WARN    beater/filebeat.go:411  Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-12-28T06:06:34.964Z        INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 0
2022-12-28T06:06:34.964Z        INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 1
2022-12-28T06:06:34.964Z        INFO    [crawler]       beater/crawler.go:117   starting input, keys present on the config: [filebeat.inputs.0.fields.evn filebeat.inputs.0.fields.language filebeat.inputs.0.fields.log-type filebeat.inputs.0.fields.server filebeat.inputs.0.fields_under_root filebeat.inputs.0.paths.0 filebeat.inputs.0.processors.0.add_kubernetes_metadata.host filebeat.inputs.0.processors.0.add_kubernetes_metadata.matchers.0.logs_path.logs_path filebeat.inputs.0.processors.1.drop_fields.fields.0 filebeat.inputs.0.processors.1.drop_fields.fields.1 filebeat.inputs.0.processors.1.drop_fields.fields.2 filebeat.inputs.0.processors.1.drop_fields.ignore_missing filebeat.inputs.0.tail_files filebeat.inputs.0.type]
2022-12-28T06:06:34.964Z        WARN    [cfgwarn]       log/input.go:89 DEPRECATED: Log input. Use Filestream input instead.
2022-12-28T06:06:34.965Z        INFO    [input] log/input.go:171        Configured paths: [/var/log/containers/*_apaas_xjsd*.log]   {"input_id": "d63f155a-8eec-4768-9026-3344c12d5a52"}
2022-12-28T06:06:34.965Z        INFO    [crawler]       beater/crawler.go:148   Starting input (ID: 870644319368240099)
2022-12-28T06:06:34.965Z        INFO    [crawler]       beater/crawler.go:106   Loading and starting Inputs completed. Enabled inputs: 1
2022-12-28T06:06:34.975Z        INFO    add_kubernetes_metadata/kubernetes.go:72        add_kubernetes_metadata: kubernetes env detected, with version: v1.20.8
2022-12-28T06:06:34.975Z        INFO    [kubernetes]    kubernetes/util.go:122  kubernetes: Using node work1 provided in the config {"libbeat.processor": "add_kubernetes_metadata"}
2022-12-28T06:06:37.953Z        INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:101    add_cloud_metadata: hosting provider type not detected.
2022-12-28T06:07:04.979Z        INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"/"},"cpuacct":{"id":"/","total":{"ns":262761106}},"memory":{"id":"/","mem":{"limit":{"bytes":209715200},"usage":{"bytes":43851776}}}},"cpu":{"system":{"ticks":50,"time":{"ms":59}},"total":{"ticks":230,"time":{"ms":250},"value":230},"user":{"ticks":180,"time":{"ms":191}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"c51f2099-815c-44b2-85cc-74b28954da22","uptime":{"ms":33068},"version":"7.17.8"},"memstats":{"gc_next":22072360,"memory_alloc":15622000,"memory_sys":43861000,"memory_total":65474720,"rss":88260608},"runtime":{"goroutines":74}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":16},"load":{"1":1.23,"15":1.36,"5":1.35,"norm":{"1":0.0769,"15":0.085,"5":0.0844}}}}}}

filebeat pod cannot open log

Hey!

Are those file maybe symlinks to other files? If so maybe you can Container input | Filebeat Reference [8.5] | Elastic to indicate to Filebeat to follow them and open the original files.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.