Filebeat filestream input with kubectl container path

Elastic Stack Beats
5

Turns out I was barking up the wrong tree.

I just realized our architecture includes an NGINX Ingress Controller, and that controller has been happily logging the Tomcat access requests and responses all along, with those logs already being collected by Filebeat.

All I had to do was add a filestream input to collect the Ingress logs, using the data_stream pattern suggested in this post

    - type: filestream
      id: ams-cache-manager-ingress-logs
      paths:
        - /var/log/containers/nginx-ingress-controller-*.log
      fields_under_root: true
      fields:
        data_stream.type: logs
        data_stream.dataset: ams
        data_stream.namespace: cache-manager-ingress
      parsers:
        - container: ~
      prospector:
        scanner:
          fingerprint.enabled: true
          symlinks: true
      file_identity.fingerprint: ~
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            namespace: ams-dev1
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"