Currently seeing an issue on linux hosts running filebeat 6.3.* in which filebeat, if stopped and restarted for any reason, is no longer able to resolve dns lookups for downstream hosts such as elasticsearch, etc.
I am able to curl and dig these hosts just fine. filebeat logs look as follows:
filebeat shuts down:
2019-01-25T04:31:48.461Z INFO [monitoring] log/log.go:133 Uptime: 10h31m3.934500227s
2019-01-25T04:31:48.461Z INFO [monitoring] log/log.go:110 Stopping metrics logging.
2019-01-25T04:31:48.461Z INFO instance/beat.go:321 filebeat stopped.
then filebeat comes back up:
2019-01-25T04:31:48.707Z INFO log/harvester.go:228 Harvester started for file: /var/log/foo.log
2019-01-25T04:31:48.711Z INFO log/harvester.go:228 Harvester started for file: /mnt/log/bar/bar.log
2019-01-25T04:31:59.533Z WARN transport/tcp.go:36 DNS lookup failure "some-domain-1": lookup
some-domain-1 on 127.0.0.1:53: read udp 127.0.0.1:40116->127.0.0.1:53: i/o timeout
2019-01-25T04:32:00.533Z ERROR pipeline/output.go:74 Failed to connect: Get
https://some-domain-1: lookup some-domain-1 on 127.0.0.1:53: read udp 127.0.0.1:40116->127.0.0.1:53: i/o timeout
I am running a local dnsmasq agent on each host running filebeat.
I attempted to substitute a binary built against libc to use the libc name-resolver instead of the pure go name-resolver but I get the same behavior.
Has anyone seen anything similar? Apologies for any typos, etc.
thanks