"stdout only" logs not captured in filebeat while display file with kubectl logs pod

Elastic Stack Beats
1

This is the filebeats config I am using via helm charts

 filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
      - type: container
        stream: all
        paths:
          - /var/log/containers/*.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}

      setup.ilm.enabled: false
      setup.template.overwrite: true

      setup.template.name: "filebeat"
      setup.template.pattern: "filebeat"
      output.elasticsearch:
        host: '${NODE_NAME}'
        hosts: '["https://${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]'
        username: '${ELASTICSEARCH_USERNAME}'
        password: '${ELASTICSEARCH_PASSWORD}'
        index: "filebeat-8.5.1-%{+yyyy.MM.dd}"
        protocol: https
        ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]

My containers output is captured in elasticsearch streams except one container which runs as below and outputs to stdout
CMD [ "npm", "run", "server" ]

There is nothing in /var/log/containers and /var/lib/ of this container. may be that is why filebeat is not able to send data?

I am able to see logs of this container via kubectl logs

I tried to skip path in filebeats.input and only read stdout but it wasn't welcomed and pods had crashloopbackoff error.

Surprisingly it is only not working in 8.5.1 but version 7 it seemed to be working. Any idea how I can capture logs of this pods/namespace in filebeats.

Much appreciated, I have spent 1 full day figuring this out but to no avail.
Also I am using containerd and I've searched everywhere on the filesystem there is no log file anywhere except stdout.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.