Kubernetes Filebeat Manifest not connecting to elasticsearch

Elastic Stack Beats
1

I followed the instructions on:
https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html

What's happening

  • The daemonset deploys successfully, the logs all seem fine. It successfully parses the configuration and loads the modules.
  • Connects to Kibana and creates dashboards and index patterns

What's not happening

  • The containers never reach out to elasticsearch. Running TCPdump on all hosts never show so much as a SYN

Debugging done / Results

  • Running filebeat -c /etc/filebeat.yml test output in one of the containers succeeds (warns about tls)
  • similar configs for packetbeat and metricbeat work perfectly as expected.

filebeat container log

2019-06-21T14:03:57.886Z	INFO	instance/beat.go:571	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-06-21T14:03:57.887Z	INFO	instance/beat.go:579	Beat ID: 961db705-0e9c-4247-bbca-085804eb1d76
2019-06-21T14:03:57.887Z	INFO	[index-management.ilm]	ilm/ilm.go:129	Policy name: filebeat-7.1.1
2019-06-21T14:03:57.888Z	INFO	[seccomp]	seccomp/seccomp.go:116	Syscall filter successfully installed
2019-06-21T14:03:57.888Z	INFO	[beat]	instance/beat.go:827	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "961db705-0e9c-4247-bbca-085804eb1d76"}}}
2019-06-21T14:03:57.888Z	INFO	[beat]	instance/beat.go:836	Build info	{"system_info": {"build": {"commit": "3358d9a5a09e3c6709a2d3aaafde628ea34e8419", "libbeat": "7.1.1", "time": "2019-05-23T13:21:33.000Z", "version": "7.1.1"}}}
2019-06-21T14:03:57.888Z	INFO	[beat]	instance/beat.go:839	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":40,"version":"go1.11.5"}}}
2019-06-21T14:03:57.907Z	INFO	[beat]	instance/beat.go:843	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-06-18T01:00:21Z","containerized":true,"name":"filebeat-2trk4","ip":["127.0.0.1/8","::1/128","172.16.149.42/32","fe80::71:29ff:fe24:e4ab/64"],"kernel_version":"3.10.0-957.12.1.el7.x86_64","mac":["02:71:29:24:e4:ab"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-06-21T14:03:57.909Z	INFO	[beat]	instance/beat.go:872	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2019-06-21T14:03:56.920Z"}}}
2019-06-21T14:03:57.909Z	INFO	instance/beat.go:280	Setup Beat: filebeat; Version: 7.1.1
2019-06-21T14:03:57.909Z	INFO	[index-management]	idxmgmt/std.go:165	Set output.elasticsearch.index to 'filebeat-7.1.1' as ILM is enabled.
2019-06-21T14:03:57.909Z	INFO	elasticsearch/client.go:165	Elasticsearch url: http://172.16.176.199:9200
2019-06-21T14:03:57.910Z	INFO	[publisher]	pipeline/module.go:97	Beat name: filebeat-2trk4
2019-06-21T14:03:57.912Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2019-06-21T14:03:57.912Z	INFO	kibana/client.go:118	Kibana url: http://kibana-core-kibana.core.svc.k8.alcazar:5601
2019-06-21T14:03:58.259Z	INFO	kibana/client.go:118	Kibana url: http://kibana-core-kibana.core.svc.k8.alcazar:5601
2019-06-21T14:04:00.888Z	INFO	add_cloud_metadata/add_cloud_metadata.go:346	add_cloud_metadata: hosting provider type not detected.
2

I think i have solved it...With some lingering confusion

I changed the docker daemon to NOT log to journald, CENTOS adds the flag by default

But this doesn't explain why filebeat wasn't at least initializing the elasticsearch connection. My theory is that during the kubernetes autodiscovery, if it doesn't at least one .log file it will NOT start up the output pipeline?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.