Filebeat Not Shipping Container logs from Kubernetes

Elastic Stack Beats
,
1

I tried the config settings for autodiscover described here:

I confirmed that the pod whose log I want to capture does produce output when I run the command kubectl logs [podName]

I also confirmed the existence of the logs (flat files) by executing kubectl exec.

However, the logs do not appear to be shipped to Elasticsearch. When I try to create an index pattern, nothing matches in Kibana Stack Management nor does any results appear in Kibana Discover under the filebeat* index pattern.

Here are the relevant sections of the filebeat-kubernetes.yaml manifest file

data:
  filebeat.yml: |-
    setup:
      template:
        name: "ams-rancher"
        pattern: "ams-rancher-%{+yyyyMMdd}"
    # filebeat.inputs:
    # - type: filestream
    #  paths:
    #    - /var/log/containers/*.log
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: filestream
            id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
            paths:
              - /opt/amsdev/logs/ams-cache-manager-logs/*.log
            parsers:
              - container: ~
                prospector:
                scanner:
                fingerprint.enabled: true
                symlinks: true
                file_identity.fingerprint: ~

    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:8.12.2
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: "{hostname}"
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value:
        - name: ELASTICSEARCH_PASSWORD
          value:
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        - name: NODE_NAME
          value: vmdev-rms4
2

Related question:

We actually have 6 nodes running in our Rancher/K8 setup. Would I enter all 6 node names into the NODE_NAME config in the manifest file if we needed to monitor all of them?

3

Hi @paolovalladolid

That article is aging a bit, conceptually still good but I would follow this

and use the suggested manifest...

https://raw.githubusercontent.com/elastic/beats/8.13/deploy/kubernetes/filebeat-kubernetes.yaml

Leave this as is.. that gets substituted at runtime.

node: ${NODE_NAME}

4

Thank you!

I renamed the manifest file I was using, and replaced it with a clean copy of the manifest file from the provided link.

The only change I made to the clean copy was to the value of ELASTICSEARCH_HOST.

So my first goal was achieved which was to ship the container log to our ES cluster.

Next is to configure the index pattern that we need to use. Will report back.

5

You should look at this cool solution I just showed another .. .Or you will need to setup up all your custom templates, naming etc..etc.. which is fine but it is a lot of work.

6

Thanks, Stephen!

In our pre-Kubernetes/Rancher days, we had been collecting logs from 6 different inputs, and so we had 6 input entries in our old filebeat.yml file.

What you advise for multiple input configuration looks similar enough, yet cleaner and more maintainable. I'll look into implementing this in our filebeat-kubernetes.yaml.

7

Whatever I'm trying to do to change the index pattern is not working.

First I tried this config wth modified lines in bold:

data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: filestream
      id: kubernetes-container-logs
      paths:
        - /var/log/containers/*.log
      # NEW LINES BELOW
      fields_under_root: true
      fields:
        data_stream.type: logs
        data_stream.dataset: ams
        data_stream.namespace: rancher
       # NEW LINES ABOVE
      parsers:
        - container: ~
      prospector:
        scanner:
          fingerprint.enabled: true
          symlinks: true
      file_identity.fingerprint: ~
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

and the ES output section

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      index: "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}"

This resulted in no logs collected, with no index pattern matching "logs-*" in Kibana.

I then tried hardcoding the index property under output.elasticsearch but that did not work either.

I then commented out the index line and was once again able to see logs in Kibana under the "filebeat-*" index pattern.

Do I need to put the setup.template lines back into my filebeat-kubernetes.yaml? That's the only thing I can think of that is breaking log collection from Kubernetes.

8

Yes I think you need, otherwise it will fail.

setup.ilm.enabled: false
setup.template.enabled: false
10

I tried modifying the processors section to try to get only logs matching a pod name. This killed the log collection.

 processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            indexers:
            - container_name: "ams-cache-manager-*"
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

I would like to configure a separate Filebeat input per container so that we can have a separate dashboard in Kibana for each container.

What is the correct way to set up one input per container?

I also tried adding a line to the fields section instead of the processors section, like this - but log collection still did not resume.

Do we need to switch to the autodiscover configuration? All examples of kubernetes.container.name being used in a manifest file only show this property under autodiscover.

data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: filestream
      id: ams-cache-manager-container-logs
      paths:
        - /var/log/containers/*.log
      fields_under_root: true
      fields:
        data_stream.type: logs
        data_stream.dataset: ams
        data_stream.namespace: rancher
        kubernetes.container.name: "ams-cache-manager-*"
      parsers:
        - container: ~
      prospector:
        scanner:
          fingerprint.enabled: true
          symlinks: true
      file_identity.fingerprint: ~
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
11

Hi @paolovalladolid Since this topic originally was solved I would recommend opening a new Topic with a New Subject and the details of your new issue.

12

Sorry about that.

I literally just now figured out how to get log collection going again. This turned out to be the way - do the matching for the desired log source in the paths not the fields:

filebeat.yml: |-
    filebeat.inputs:
    - type: filestream
      id: ams-cache-manager-container-logs
      paths:
        - /var/log/containers/ams-cache*.log
      fields_under_root: true
      fields:
        data_stream.type: logs
        data_stream.dataset: ams
        data_stream.namespace: rancher

Thanks for all the help!