No logs shipped from Kubernetes Filebeat

jmgirven · January 21, 2020, 2:45pm

I am running filebeats 7.5.1 on my kubernetes cluster as described in the section "Get started in minutes" "Hosted" "Kubernetes Logs" from here:

https://www.elastic.co/what-is/kubernetes-monitoring

With the exception that I am using the filebeat-kubernetes.yaml for version 7.5.1 instead of the referenced 6.0 file (which I discovered here).

I am trying to ship my pod logs to my hosted elastic.co cluster. The filebeat deamonset has all the pods running, and there are no obvious error logs, but I am not seeing any logs in the elastic.co Kibana logs dashboard. What am I missing?

jsoriano · January 21, 2020, 4:42pm

Hi @jmgirven and welcome to discuss!

Oops, sorry! we should update this link.

How is your Kubernetes cluster deployed? Does it have something special like the use of some logging driver?

Could you share the startup logs of one of your filebeat pods?

jmgirven · January 21, 2020, 5:18pm

Our Kubernetes cluster is a managed service running on Oracle Container Engine for Kubernetes. I don't believe it has a special logging driver, but it might. Is there a e.g. kubectl command to check?

Sure, does this help:

2020-01-21T14:13:04.849Z INFO instance/beat.go:610 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-01-21T14:13:04.849Z INFO instance/beat.go:618 Beat ID: 6a60c02e-4e8f-4f65-b236-004b8b1be79e
2020-01-21T14:13:04.855Z INFO add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-01-21T14:13:04.856Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-01-21T14:13:04.856Z INFO [beat] instance/beat.go:941 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "6a60c02e-4e8f-4f65-b236-004b8b1be79e"}}}
2020-01-21T14:13:04.856Z INFO [beat] instance/beat.go:950 Build info {"system_info": {"build": {"commit": "60dd883ca29e1fdd5b8b075bd5f3698948b1d44d", "libbeat": "7.5.1", "time": "2019-12-16T21:56:14.000Z", "version": "7.5.1"}}}
2020-01-21T14:13:04.856Z INFO [beat] instance/beat.go:953 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.12"}}}
2020-01-21T14:13:04.859Z INFO [beat] instance/beat.go:957 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-14T10:22:07Z","containerized":false,"name":"oke-c4tinjsgjsd-n2tiyjsgq4d-sj5pd2epukq-1","ip":["127.0.0.1/8","10.0.10.8/24","172.17.0.1/16","10.244.4.0/32","10.244.4.1/24"],"kernel_version":"4.14.35-1902.8.4.el7uek.x86_64","mac":["02:00:17:01:e2:ff","02:42:2a:56:83:0f","fe:46:f0:73:f5:df","4e:26:82:f4:b3:e2","7a:7c:03:50:66:da","ae:5e:5c:d7:d4:3e","c2:6a:3e:96:78:d6","42:3b:85:53:a2:c9","6a:ad:09:33:09:6a","56:0a:0c:35:82:3d","76:25:24:e1:bb:24","6e:c3:e8:eb:8b:f1","a2:92:41:1c:7c:fc","7e:3b:e8:a2:1c:70","fe:86:c9:09:66:23","0e:e6:f9:a6:b0:ed","8e:ec:49:a4:e8:62","9a:0a:53:2d:12:70","16:df:85:53:2f:8f","ba:6a:5c:ff:1c:8f","02:38:13:4a:72:3d","0e:a8:50:1a:d1:2e","8e:63:52:2a:c0:a4","56:25:61:93:47:ea"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-01-21T14:13:04.859Z INFO [beat] instance/beat.go:986 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-01-21T14:13:04.290Z"}}}
2020-01-21T14:13:04.860Z INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.5.1
2020-01-21T14:13:04.860Z INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'filebeat-7.5.1' as ILM is enabled.
2020-01-21T14:13:04.860Z INFO elasticsearch/client.go:171 Elasticsearch url: https://33dd2e19ea264ae3ac3cb95ee80c6d1c.europe-west2.gcp.elastic-cloud.com:443
2020-01-21T14:13:04.860Z INFO [publisher] pipeline/module.go:97 Beat name: oke-c4tinjsgjsd-n2tiyjsgq4d-sj5pd2epukq-1
2020-01-21T14:13:04.861Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-01-21T14:13:04.861Z INFO instance/beat.go:429 filebeat start running.
2020-01-21T14:13:04.862Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-01-21T14:13:04.862Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-01-21T14:13:04.862Z INFO crawler/crawler.go:72 Loading Inputs: 1
2020-01-21T14:13:04.873Z INFO add_kubernetes_metadata/kubernetes.go:68 add_kubernetes_metadata: kubernetes env detected, with version: v1.14.8
2020-01-21T14:13:04.873Z INFO kubernetes/util.go:79 kubernetes: Using node 10.0.10.8 provided in the config
2020-01-21T14:13:04.974Z INFO log/input.go:152 Configured paths: [/var/log/containers/*.log]
2020-01-21T14:13:04.974Z INFO input/input.go:114 Starting input of type: container; ID: 3960813757973143954
2020-01-21T14:13:04.974Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-01-21T14:13:34.864Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":50}},"total":{"ticks":160,"time":{"ms":171},"value":160},"user":{"ticks":120,"time":{"ms":121}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"c2be0d79-bb4e-4544-9d94-a72d6dbfd4b5","uptime":{"ms":30041}},"memstats":{"gc_next":10911632,"memory_alloc":5629496,"memory_total":16939904,"rss":55189504},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4},"load":{"1":0.06,"15":0.31,"5":0.17,"norm":{"1":0.015,"15":0.0775,"5":0.0425}}}}}}
2020-01-21T14:14:04.864Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":9}},"total":{"ticks":180,"time":{"ms":22},"value":180},"user":{"ticks":130,"time":{"ms":13}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"c2be0d79-bb4e-4544-9d94-a72d6dbfd4b5","uptime":{"ms":60041}},"memstats":{"gc_next":10911632,"memory_alloc":6297736,"memory_total":17608144},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.17,"15":0.31,"5":0.18,"norm":{"1":0.0425,"15":0.0775,"5":0.045}}}}}}
2020-01-21T14:14:34.864Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":11}},"total":{"ticks":200,"time":{"ms":23},"value":200},"user":{"ticks":140,"time":{"ms":12}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"c2be0d79-bb4e-4544-9d94-a72d6dbfd4b5","uptime":{"ms":90041}},"memstats":{"gc_next":10911632,"memory_alloc":6760848,"memory_total":18071256},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.1,"15":0.3,"5":0.17,"norm":{"1":0.025,"15":0.075,"5":0.0425}}}}}}
2020-01-21T14:15:04.864Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":10}},"total":{"ticks":230,"time":{"ms":22},"value":230},"user":{"ticks":150,"time":{"ms":12}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"c2be0d79-bb4e-4544-9d94-a72d6dbfd4b5","uptime":{"ms":120041}},"memstats":{"gc_next":10911632,"memory_alloc":7309504,"memory_total":18619912},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.14,"15":0.29,"5":0.16,"norm":{"1":0.035,"15":0.0725,"5":0.04}}}}}}

@jsoriano

jsoriano · January 21, 2020, 6:22pm

Could you check if the docker logs in your machine are under /var/log/containers?

If you cannot find your logs you can try to login into one of your kubernetes nodes and use docker inspect to check the configuration and the log path of one of your running containers.

Something else you can also try is to use Beats autodiscover, for that follow the instructions in the manifest you downloaded.

jmgirven · January 22, 2020, 10:16am

If I list the /var/log/containers directory mounted into one of the filebeat containers, e.g:

$ kubectl -n kube-system exec -it filebeat-xyzab -- ls -l /var/log/containers/

I can see the many log files for my containers there. The files are all symlinks to e.g: /var/log/pods/namespace_podname-xyzcodezyx/podname/0.log, which seems fine because /var/log is mounted as a volume into the filebeat container. However this is also a symlink in my case, to e.g: /u01/data/docker/containers/xyzxyz/zyxzyx-json.log, which is not a filebeat volume.

I have copied the volume mount configuration for /var/lib/docker/containers except replaced it with /u01/data/docker/containers, and recreated my filebeat daemonset. Now the logs are coming through to elastic.co, thanks. I assume this is just some specific configuration to OKE.

jsoriano · January 22, 2020, 11:48am

Great, thanks for posting the solution you have found!

Vinicius_Maia · February 10, 2020, 7:40pm

Hi jmgirven, I replaced the config that you comment (/u01/data/docker/containers) but I didn't have success.
I'm using OKE as you and my elasticsearch don't receive filebeat index.

Following, my filebeat.yaml

---

apiVersion: v1

kind: ConfigMap

metadata:

  name: filebeat-config

  namespace: elk

  labels:

    k8s-app: filebeat

data:

  filebeat.yml: |-

    filebeat.inputs:

    - type: container

      paths:

        - /u01/data/docker/containers/*.log

      processors:

        - add_kubernetes_metadata:

            host: ${NODE_NAME}

            matchers:

            - logs_path:

                logs_path: "/u01/data/docker/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:

    #filebeat.autodiscover:

    #  providers:

    #    - type: kubernetes

    #      host: ${NODE_NAME}

    #      hints.enabled: true

    #      hints.default_config:

    #        type: container

    #        paths:

    #          - /u01/data/docker/containers/*${data.kubernetes.container.id}.log

    processors:

      - add_cloud_metadata:

      - add_host_metadata:

    output.elasticsearch:

      hosts: ['${ELASTICSEARCH_HOST:MY_IP_ADDRESS}:${ELASTICSEARCH_PORT:9200}']

---

apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: filebeat

  namespace: elk

  labels:

    k8s-app: filebeat

spec:

  selector:

    matchLabels:

      k8s-app: filebeat

  template:

    metadata:

      labels:

        k8s-app: filebeat

    spec:

      serviceAccountName: filebeat

      terminationGracePeriodSeconds: 30

      hostNetwork: true

      dnsPolicy: ClusterFirstWithHostNet

      containers:

      - name: filebeat

        image: docker.elastic.co/beats/filebeat:7.5.2

        args: [

          "-c", "/etc/filebeat.yml",

          "-e",

        ]

        env:

        - name: ELASTICSEARCH_HOST

          value: 10.2.0.22

        - name: ELASTICSEARCH_PORT

          value: "9200"

        - name: NODE_NAME

          valueFrom:

            fieldRef:

              fieldPath: spec.nodeName

        securityContext:

          runAsUser: 0

        resources:

          limits:

            memory: 200Mi

          requests:

            cpu: 100m

            memory: 100Mi

        volumeMounts:

        - name: config

          mountPath: /etc/filebeat.yml

          readOnly: true

          subPath: filebeat.yml

        - name: data

          mountPath: /usr/share/filebeat/data

        - name: varlibdockercontainers

          mountPath: /var/lib/docker/containers

          readOnly: true

        - name: varlog

          mountPath: /var/log

          readOnly: true

      volumes:

      - name: config

        configMap:

          defaultMode: 0600

          name: filebeat-config

      - name: varlibdockercontainers

        hostPath:

          path: /var/lib/docker/containers

      - name: varlog

        hostPath:

          path: /var/log

      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart

      - name: data

        hostPath:

          path: /var/lib/filebeat-data

          type: DirectoryOrCreate

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: filebeat

subjects:

- kind: ServiceAccount

  name: filebeat

  namespace: elk

roleRef:

  kind: ClusterRole

  name: filebeat

  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  name: filebeat

  labels:

    k8s-app: filebeat

rules:

- apiGroups: [""] # "" indicates the core API group

  resources:

  - namespaces

  - pods

  verbs:

  - get

  - watch

  - list

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: filebeat

  namespace: elk

  labels:

    k8s-app: filebeat

---

Vinicius_Maia · February 11, 2020, 1:34pm

Finally I fixed my setup and have filebeat sending data to Elasticsearch.
I had mount in my OKE, the path: /u01/data/docker in the volumeMounts and volumes in the name: varlibdockercontainers variables.
Thanks to @jmgirven for the tip.

system · March 10, 2020, 1:34pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.