so for further context, I have a three node elastic cluster. I have kibana running as well. I showed es02's config for the context of the autodiscovery hints. es01 is my primary node. The filebeat logs don't appear to show any errors, that I can tell. I can see the filebeat index in Kibana and it does appear documents are being sent to it. However, when I go to discover or logs I don't see anything for my es02 container.
Attaching to filebeat
filebeat | 2021-04-08T03:08:15.066Z INFO instance/beat.go:660 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
filebeat | 2021-04-08T03:08:15.070Z INFO instance/beat.go:668 Beat ID: 7d967ca0-addd-4138-ae73-f964417e4153
filebeat | 2021-04-08T03:08:15.073Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
filebeat | 2021-04-08T03:08:15.089Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
filebeat | 2021-04-08T03:08:15.089Z INFO [beat] instance/beat.go:996 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "7d967ca0-addd-4138-ae73-f964417e4153"}}}
filebeat | 2021-04-08T03:08:15.089Z INFO [beat] instance/beat.go:1005 Build info {"system_info": {"build": {"commit": "08e20483a651ea5ad60115f68ff0e53e6360573a", "libbeat": "7.12.0", "time": "2021-03-18T06:16:51.000Z", "version": "7.12.0"}}}
filebeat | 2021-04-08T03:08:15.089Z INFO [beat] instance/beat.go:1008 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":12,"version":"go1.15.8"}}}
filebeat | 2021-04-08T03:08:15.090Z INFO [beat] instance/beat.go:1012 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-04-07T21:37:24Z","containerized":true,"name":"726d12a12558","ip":["127.0.0.1/8","172.25.0.13/24"],"kernel_version":"4.19.104-microsoft-standard","mac":["02:42:ac:19:00:0d"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"44ccc339d95bd51386fcfc5d8f041927"}}}
filebeat | 2021-04-08T03:08:15.090Z INFO [beat] instance/beat.go:1041 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 8, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-04-08T03:08:14.600Z"}}}
filebeat | 2021-04-08T03:08:15.090Z INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.12.0
filebeat | 2021-04-08T03:08:15.091Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.12.0' as ILM is enabled.
filebeat | 2021-04-08T03:08:15.091Z WARN [cfgwarn] tlscommon/config.go:101 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
filebeat | 2021-04-08T03:08:15.092Z INFO eslegclient/connection.go:99 elasticsearch url: https://es01:9200
filebeat | 2021-04-08T03:08:15.093Z INFO [publisher] pipeline/module.go:113 Beat name: 726d12a12558
filebeat | 2021-04-08T03:08:15.098Z INFO beater/filebeat.go:117 Enabled modules/filesets: kibana (audit, log), (), system (auth, syslog), elasticsearch (audit, deprecation, gc, server, slowlog)
filebeat | 2021-04-08T03:08:15.101Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
filebeat | 2021-04-08T03:08:15.101Z INFO instance/beat.go:468 filebeat start running.
filebeat | 2021-04-08T03:08:15.104Z INFO memlog/store.go:119 Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
filebeat | 2021-04-08T03:08:15.104Z INFO memlog/store.go:124 Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
filebeat | 2021-04-08T03:08:15.104Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
filebeat | 2021-04-08T03:08:15.104Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 10
filebeat | 2021-04-08T03:08:15.104Z WARN [cfgwarn] docker/input.go:49 DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
filebeat | 2021-04-08T03:08:15.105Z INFO log/input.go:157 Configured paths: [/var/lib/docker/containers/*/*.log]
filebeat | 2021-04-08T03:08:15.105Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 2482010575008624874)
filebeat | 2021-04-08T03:08:15.105Z INFO log/input.go:157 Configured paths: [/var/log/auth.log* /var/log/secure*]
filebeat | 2021-04-08T03:08:15.105Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 11991761552713575762)
filebeat | 2021-04-08T03:08:15.106Z INFO log/input.go:157 Configured paths: [/var/log/messages* /var/log/syslog*]
filebeat | 2021-04-08T03:08:15.106Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 7521169057727086769)
filebeat | 2021-04-08T03:08:15.106Z INFO log/input.go:157 Configured paths: [/var/log/elasticsearch/*.log /var/log/elasticsearch/*_server.json]
filebeat | 2021-04-08T03:08:15.106Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 10584868271925815388)
filebeat | 2021-04-08T03:08:15.106Z INFO log/input.go:157 Configured paths: [/var/log/elasticsearch/*_index_search_slowlog.log /var/log/elasticsearch/*_index_indexing_slowlog.log /var/log/elasticsearch/*_index_search_slowlog.json /var/log/elasticsearch/*_index_indexing_slowlog.json]
filebeat | 2021-04-08T03:08:15.107Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 6878571439949832352)
filebeat | 2021-04-08T03:08:15.108Z INFO log/input.go:157 Configured paths: [/var/log/elasticsearch/*_access.log /var/log/elasticsearch/*_audit.log /var/log/elasticsearch/*_audit.json]
filebeat | 2021-04-08T03:08:15.108Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 2322925157510133784)
filebeat | 2021-04-08T03:08:15.109Z INFO log/input.go:157 Configured paths: [/var/log/elasticsearch/*_deprecation.log /var/log/elasticsearch/*_deprecation.json]
filebeat | 2021-04-08T03:08:15.109Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 14614850079161566516)
filebeat | 2021-04-08T03:08:15.109Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/06c4421b670d776330c7e18b767afb1f615f9498bbd1a49a46619e4d9d715ec4/06c4421b670d776330c7e18b767afb1f615f9498bbd1a49a46619e4d9d715ec4-json.log
filebeat | 2021-04-08T03:08:15.109Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/723676c36414ac323759a60c14906c626b9c0e539e05f626ae9fc41c1d095606/723676c36414ac323759a60c14906c626b9c0e539e05f626ae9fc41c1d095606-json.log
filebeat | 2021-04-08T03:08:15.109Z INFO log/input.go:157 Configured paths: [/var/log/elasticsearch/gc.log.[0-9]* /var/log/elasticsearch/gc.log]
filebeat | 2021-04-08T03:08:15.109Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 11922003923008771421)
filebeat | 2021-04-08T03:08:15.109Z INFO log/input.go:157 Configured paths: [/var/log/kibana/*_audit.json]
filebeat | 2021-04-08T03:08:15.109Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 14651632209134346)
filebeat | 2021-04-08T03:08:15.109Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/72ecf3ae0802d69af5225c2e648fb110c8a5df16ad7533bf3ee45e3c4c8d2f91/72ecf3ae0802d69af5225c2e648fb110c8a5df16ad7533bf3ee45e3c4c8d2f91-json.log
filebeat | 2021-04-08T03:08:15.110Z INFO log/input.go:157 Configured paths: [/var/log/kibana/kibana.stdout]
filebeat | 2021-04-08T03:08:15.110Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 9913943773415849758)
filebeat | 2021-04-08T03:08:15.110Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/7ab3e7c8ca7333649306bb7e29349c778e26d4e5032dbf714776ceeaf342d766/7ab3e7c8ca7333649306bb7e29349c778e26d4e5032dbf714776ceeaf342d766-json.log
filebeat | 2021-04-08T03:08:15.110Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/353a91c46929e5590f50b40d37abc658579035d34ec5d64f296282a6d714fa2f/353a91c46929e5590f50b40d37abc658579035d34ec5d64f296282a6d714fa2f-json.log
filebeat | 2021-04-08T03:08:15.110Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/710c24f7146fd91ab1b611e8b1aa7bdb598fd4980f86d23bfe807546615f2b9e/710c24f7146fd91ab1b611e8b1aa7bdb598fd4980f86d23bfe807546615f2b9e-json.log
filebeat | 2021-04-08T03:08:15.110Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 10
filebeat | 2021-04-08T03:08:15.110Z INFO cfgfile/reload.go:164 Config reloader started
filebeat | 2021-04-08T03:08:15.110Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/c21a40bd2e179b6e23669ed165964500ff476e95ad3c632f3b3089141754d96b/c21a40bd2e179b6e23669ed165964500ff476e95ad3c632f3b3089141754d96b-json.log
filebeat | 2021-04-08T03:08:15.110Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/8a4e3b8bbeb4736f31e527d61da97f52cab9b5a04f6912059cd3ae8220c5db04/8a4e3b8bbeb4736f31e527d61da97f52cab9b5a04f6912059cd3ae8220c5db04-json.log
filebeat | 2021-04-08T03:08:15.110Z INFO cfgfile/reload.go:224 Loading of config files completed.
filebeat | 2021-04-08T03:08:15.110Z INFO log/harvester.go:302 Harvester started for file: /var/lib/docker/containers/c4e7e39e3f1feb5ec7f6249e5d6b9e1d48455e02f11529888fd01fdef6f24970/c4e7e39e3f1feb5ec7f6249e5d6b9e1d48455e02f11529888fd01fdef6f24970-json.log