Filebeat filestream input with kubectl container path

Elastic Stack Beats
1

I am using a slightly modified version of the manifest file at the official Filebeat and Kubernetes page. It is working fine with multiple filestream inputs.

Now I am trying to add a filestream input to collect Tomcat access logs. I can see them when I run this command on the shell:

kubectl exec ams-cache-manager-dev1-646fff567f-xb4x6 -n ams-dev1 -- ls opt/amsdev/logs/ams-cache-manager-logs

However I don't see them being collected along with the other logs when I look in Kibana Discover. This is the config for that input.

    - type: filestream
      id: ams-cache-manager-tomcat-container-logs
      paths:
        - /opt/amsdev/logs/ams-cache-manager-logs/*.log
      fields_under_root: true
      fields:
        data_stream.type: logs
        data_stream.dataset: ams
        data_stream.namespace: cache-manager-tomcat
      parsers:
        - container: ~
      prospector:
        scanner:
          fingerprint.enabled: true
          symlinks: true
      file_identity.fingerprint: ~
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            namespace: ams-dev1
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

Filebeat was collecting logs for this input earlier under this config but tomcat logs were also not included

 - type: filestream
      id: ams-cache-manager-tomcat-container-logs
      paths:
        - /var/log/containers/*.log
      fields_under_root: true
      fields:
        data_stream.type: logs
        data_stream.dataset: ams
        data_stream.namespace: cache-manager-tomcat
      parsers:
        - container: ~
      prospector:
        scanner:
          fingerprint.enabled: true
          symlinks: true
      file_identity.fingerprint: ~
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            namespace: ams-dev1
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

What is the correct config so that Filebeat will collect the Tomcat logs listed under opt/amsdev/logs/ams-cache-manager-log as found by kubectl exec?

2

Looks like the problem is Tomcat is not logging to stdout and that is why the Tomcat access log is not showing up under /var/log/containers

And thus, Filebeat cannot see the Tomcat logs. Unless there is another Filebeat input that i'm unaware of. We're using filestream input which seems to grab anything that drops into /var/log/containers. Tried autodiscovery before but never got it to work.

3

I should note that what we are running in K8 is a Spring Boot application, which has an embedded Tomcat server.

One possible cause of our problem is the Tomcat access log is not showing when we run kubectl log. I realize this might be a K8 problem and not a Filebeat problem but I'd appreciate hints/tips from those of you who have encountered a similar problem.

4

I got Tomcat access requests and responses to show up in the console output by implementing the solution posted by JohanB here:

But there is still a disconnect between the console output, Kubernetes/Rancher, and Filebeat. The HTTP request and response in particular is still not being collected into Elasticsearch.

I'll try adding a file appender to the config in the newly added logback-access.xml file and see if that gets picked up by K8

5

Turns out I was barking up the wrong tree.

I just realized our architecture includes an NGINX Ingress Controller, and that controller has been happily logging the Tomcat access requests and responses all along, with those logs already being collected by Filebeat.

All I had to do was add a filestream input to collect the Ingress logs, using the data_stream pattern suggested in this post

    - type: filestream
      id: ams-cache-manager-ingress-logs
      paths:
        - /var/log/containers/nginx-ingress-controller-*.log
      fields_under_root: true
      fields:
        data_stream.type: logs
        data_stream.dataset: ams
        data_stream.namespace: cache-manager-ingress
      parsers:
        - container: ~
      prospector:
        scanner:
          fingerprint.enabled: true
          symlinks: true
      file_identity.fingerprint: ~
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            namespace: ams-dev1
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"