I'm trying to use filebeat in my k8s cluster to forward logs to a dedicated self-hosted elasticsearch instance.
Following the documentation here ( https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html ) I deployed the necessary k8s resources into my kube-system namespace.
The deployed PODs on the k8s nodes constantly CrashLoopBackOff and this is what I get from one of them and I have no clue what the problem is :
$ kubectl -n kube-system logs filebeat-72qph
2019-10-29T10:46:22.778Z INFO instance/beat.go:607 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-10-29T10:46:22.778Z INFO instance/beat.go:615 Beat ID: a0bac015-369a-4076-885c-fea79ba2fa80
2019-10-29T10:46:22.780Z INFO add_cloud_metadata/add_cloud_metadata.go:87 add_cloud_metadata: hosting provider type not detected.
2019-10-29T10:46:22.782Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2019-10-29T10:46:22.782Z INFO [beat] instance/beat.go:903 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "a0bac015-369a-4076-885c-fea79ba2fa80"}}}
2019-10-29T10:46:22.782Z INFO [beat] instance/beat.go:912 Build info {"system_info": {"build": {"commit": "12ee6cd05c1bfdc69721ddab1f473417b1514576", "libbeat": "7.4.1", "time": "2019-10-22T16:22:37.000Z", "version": "7.4.1"}}}
2019-10-29T10:46:22.782Z INFO [beat] instance/beat.go:915 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.9"}}}
2019-10-29T10:46:22.784Z INFO [beat] instance/beat.go:919 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-10-01T08:16:20Z","containerized":false,"name":"shinefour-rancher-agent-2","ip":["127.0.0.1/8","::1/128","116.203.87.129/32","2a01:4f8:c2c:3c0a::1/64","fe80::9400:ff:fe31:d214/64","10.0.0.4/32","fe80::8400:ff:fe31:d6ee/64","172.17.0.1/16","fe80::42:5eff:fe53:c773/64","10.42.1.0/32","fe80::6069:9eff:fe1d:b240/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64","fe80::ecee:eeff:feee:eeee/64"],"kernel_version":"4.15.0-64-generic","mac":["96:00:00:31:d2:14","86:00:00:31:d6:ee","02:42:5e:53:c7:73","62:69:9e:1d:b2:40","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-10-29T10:46:22.784Z INFO [beat] instance/beat.go:948 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-10-29T10:46:21.980Z"}}}
2019-10-29T10:46:22.785Z INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.4.1
2019-10-29T10:46:22.785Z INFO [index-management] idxmgmt/std.go:178 Set output.elasticsearch.index to 'filebeat-7.4.1' as ILM is enabled.
2019-10-29T10:46:22.785Z INFO elasticsearch/client.go:170 Elasticsearch url: https://es.rancher.shinefour.de:9200
2019-10-29T10:46:22.785Z INFO [publisher] pipeline/module.go:97 Beat name: shinefour-rancher-agent-2
2019-10-29T10:46:22.786Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2019-10-29T10:46:22.786Z INFO instance/beat.go:422 filebeat start running.
2019-10-29T10:46:22.786Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2019-10-29T10:46:22.786Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2019-10-29T10:46:22.786Z INFO crawler/crawler.go:72 Loading Inputs: 1
2019-10-29T10:46:22.788Z INFO kubernetes/util.go:67 kubernetes: Using node agent-2 provided in the config
2019-10-29T10:46:22.889Z INFO log/input.go:152 Configured paths: [/var/log/containers/*.log]
2019-10-29T10:46:22.889Z INFO input/input.go:114 Starting input of type: container; ID: 18315498249972727592
2019-10-29T10:46:22.889Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
E1029 10:46:22.889366 1 runtime.go:73] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Any help identifying the problem appreciated !
