I followed the instructions on:
https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html
What's happening
- The daemonset deploys successfully, the logs all seem fine. It successfully parses the configuration and loads the modules.
- Connects to Kibana and creates dashboards and index patterns
What's not happening
- The containers never reach out to elasticsearch. Running TCPdump on all hosts never show so much as a SYN
Debugging done / Results
- Running
filebeat -c /etc/filebeat.yml test outputin one of the containers succeeds (warns about tls) - similar configs for packetbeat and metricbeat work perfectly as expected.
filebeat container log
2019-06-21T14:03:57.886Z INFO instance/beat.go:571 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-06-21T14:03:57.887Z INFO instance/beat.go:579 Beat ID: 961db705-0e9c-4247-bbca-085804eb1d76
2019-06-21T14:03:57.887Z INFO [index-management.ilm] ilm/ilm.go:129 Policy name: filebeat-7.1.1
2019-06-21T14:03:57.888Z INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
2019-06-21T14:03:57.888Z INFO [beat] instance/beat.go:827 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "961db705-0e9c-4247-bbca-085804eb1d76"}}}
2019-06-21T14:03:57.888Z INFO [beat] instance/beat.go:836 Build info {"system_info": {"build": {"commit": "3358d9a5a09e3c6709a2d3aaafde628ea34e8419", "libbeat": "7.1.1", "time": "2019-05-23T13:21:33.000Z", "version": "7.1.1"}}}
2019-06-21T14:03:57.888Z INFO [beat] instance/beat.go:839 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":40,"version":"go1.11.5"}}}
2019-06-21T14:03:57.907Z INFO [beat] instance/beat.go:843 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-06-18T01:00:21Z","containerized":true,"name":"filebeat-2trk4","ip":["127.0.0.1/8","::1/128","172.16.149.42/32","fe80::71:29ff:fe24:e4ab/64"],"kernel_version":"3.10.0-957.12.1.el7.x86_64","mac":["02:71:29:24:e4:ab"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-06-21T14:03:57.909Z INFO [beat] instance/beat.go:872 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2019-06-21T14:03:56.920Z"}}}
2019-06-21T14:03:57.909Z INFO instance/beat.go:280 Setup Beat: filebeat; Version: 7.1.1
2019-06-21T14:03:57.909Z INFO [index-management] idxmgmt/std.go:165 Set output.elasticsearch.index to 'filebeat-7.1.1' as ILM is enabled.
2019-06-21T14:03:57.909Z INFO elasticsearch/client.go:165 Elasticsearch url: http://172.16.176.199:9200
2019-06-21T14:03:57.910Z INFO [publisher] pipeline/module.go:97 Beat name: filebeat-2trk4
2019-06-21T14:03:57.912Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2019-06-21T14:03:57.912Z INFO kibana/client.go:118 Kibana url: http://kibana-core-kibana.core.svc.k8.alcazar:5601
2019-06-21T14:03:58.259Z INFO kibana/client.go:118 Kibana url: http://kibana-core-kibana.core.svc.k8.alcazar:5601
2019-06-21T14:04:00.888Z INFO add_cloud_metadata/add_cloud_metadata.go:346 add_cloud_metadata: hosting provider type not detected.