The rumors of my giving up this post were highly overrated. 
Made a few changes to my setup and managed to once again reproduce the problem. Formerly i'd done this with manually run binaries from downloaded archives. Then i moved on to a docker setup (full ELK stack in docker with fleet server running on the host OS). And now this...
My new setup is as follows:
- Dedicated ubuntu server
- Elasticsearch, Kibana, Logstash, Fleet Server in v.8.1.3 (the version is purposeful as i want to test an upgrade to 8.2+) and installed from official DEB packages.
- Single-node, nothing fancy
- No agents connected to the fleet server as of yet, only beats (specifically Packetbeat and Winlogbeat)
I still haven't seen any direct evidence of a connection error in the logs even with debug mode activated, but that may be due to my lack of experience with the product. So here is the output from a working packetbeat connection:
{"log.level":"info","@timestamp":"2022-05-17T12:47:10.839-0400","log.origin":{"file.name":"instance/beat.go","file.line":669},"message":"Home path: [C:\\home\\j\\beats\\packetbeat-8.1.3] Config path: [C:\\home\\j\\beats\\packetbeat-8.1.3] Data path: [C:\\home\\j\\beats\\packetbeat-8.1.3\\data] Logs path: [C:\\home\\j\\beats\\packetbeat-8.1.3\\logs]","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:10.846-0400","log.origin":{"file.name":"instance/beat.go","file.line":677},"message":"Beat ID: abeb4cc3-a5b5-4665-9625-973990d2d259","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T12:47:13.858-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.877-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1047},"message":"Beat info","service.name":"packetbeat","system_info":{"beat":{"path":{"config":"C:\\home\\j\\beats\\packetbeat-8.1.3","data":"C:\\home\\j\\beats\\packetbeat-8.1.3\\data","home":"C:\\home\\j\\beats\\packetbeat-8.1.3","logs":"C:\\home\\j\\beats\\packetbeat-8.1.3\\logs"},"type":"packetbeat","uuid":"abeb4cc3-a5b5-4665-9625-973990d2d259"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.882-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1056},"message":"Build info","service.name":"packetbeat","system_info":{"build":{"commit":"271435c21bfd4e2e621d87c04f4b815980626978","libbeat":"8.1.3","time":"2022-04-19T09:56:30.000Z","version":"8.1.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.882-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1059},"message":"Go runtime info","service.name":"packetbeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":1,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.885-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Host info","service.name":"packetbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-17T11:56:29.8-04:00","name":"win10client","ip":["fe80::9cf6:869c:8fb2:709b/64","WORKING_CLIENT_IP/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.19041.1706 (WinBuild.160101.0800)","mac":["08:00:27:ed:6f:6a"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"19044.1706"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"ce58b856-23f4-4c56-8def-ee43ed3ae2d3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.886-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Process info","service.name":"packetbeat","system_info":{"process":{"cwd":"C:\\home\\j\\beats\\packetbeat-8.1.3","exe":"C:\\home\\j\\beats\\packetbeat-8.1.3\\packetbeat.exe","name":"packetbeat.exe","pid":2360,"ppid":5016,"start_time":"2022-05-17T12:47:10.690-0400"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.886-0400","log.origin":{"file.name":"instance/beat.go","file.line":323},"message":"Setup Beat: packetbeat; Version: 8.1.3","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.900-0400","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: win10client","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.903-0400","log.logger":"npcap","log.origin":{"file.name":"beater/install_npcap.go","file.line":49},"message":"npcap version: Npcap version 1.60, based on libpcap version 1.10.2-PRE-GIT","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.908-0400","log.origin":{"file.name":"procs/procs.go","file.line":103},"message":"Process watcher disabled","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T12:47:13.927-0400","log.logger":"cfgwarn","log.origin":{"file.name":"sip/plugin.go","file.line":67},"message":"BETA: packetbeat SIP protocol is used","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.944-0400","log.origin":{"file.name":"sniffer/device.go","file.line":98},"message":"Resolved device index 0 to device: \\Device\\NPF_{14FB7DA4-DA62-4E32-BFF4-92C5A25C40D1}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.948-0400","log.origin":{"file.name":"instance/beat.go","file.line":489},"message":"packetbeat start running.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.949-0400","log.origin":{"file.name":"procs/procs.go","file.line":103},"message":"Process watcher disabled","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T12:47:13.951-0400","log.logger":"cfgwarn","log.origin":{"file.name":"sip/plugin.go","file.line":67},"message":"BETA: packetbeat SIP protocol is used","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.964-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.969-0400","log.origin":{"file.name":"sniffer/device.go","file.line":98},"message":"Resolved device index 0 to device: \\Device\\NPF_{14FB7DA4-DA62-4E32-BFF4-92C5A25C40D1}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:16.882-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":101},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:31.023-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(async(tcp://MY_SERVER_IP:5044))","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:31.027-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(async(tcp://MY_SERVER_IP:5044)) established","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:43.984-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"packetbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":484,"time":{"ms":484}},"total":{"ticks":499,"time":{"ms":499},"value":499},"user":{"ticks":15,"time":{"ms":15}}},"handles":{"open":189},"info":{"ephemeral_id":"59cb2633-8bdc-4c88-8a1d-236fa1d8e853","uptime":{"ms":33205},"version":"8.1.3"},"memstats":{"gc_next":11707680,"memory_alloc":6820360,"memory_sys":22929992,"memory_total":23320688,"rss":52240384},"runtime":{"goroutines":55}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":23,"active":0,"batches":4,"total":23},"read":{"bytes":24},"type":"logstash","write":{"bytes":12808}},"pipeline":{"clients":30,"events":{"active":0,"published":23,"retry":8,"total":23},"queue":{"acked":23,"max_events":4096}}},"system":{"cpu":{"cores":1}}},"ecs.version":"1.6.0"}}
Please notice the two lines where i replaced my server ip with "MY_SERVER_IP".
{"log.level":"info","@timestamp":"2022-05-17T12:47:31.023-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(async(tcp://MY_SERVER_IP:5044))","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:31.027-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(async(tcp://MY_SERVER_IP:5044)) established","service.name":"packetbeat","ecs.version":"1.6.0"}
Aside from the obvious -- i.e. a brand new index in my kibana -- it's my indication that a successful connection was established.
Now here's the same output from a different client -- no connection even though the day before it did and generated a new index.
{"log.level":"info","@timestamp":"2022-05-17T11:52:31.292-0400","log.origin":{"file.name":"instance/beat.go","file.line":669},"message":"Home path: [C:\\home\\j\\beats\\packetbeat-8.1.3] Config path: [C:\\home\\j\\beats\\packetbeat-8.1.3] Data path: [C:\\home\\j\\beats\\packetbeat-8.1.3\\data] Logs path: [C:\\home\\j\\beats\\packetbeat-8.1.3\\logs]","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:31.293-0400","log.origin":{"file.name":"instance/beat.go","file.line":677},"message":"Beat ID: 0b0480b3-ad86-4621-897d-7ba6fc829090","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T11:52:34.303-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.308-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1047},"message":"Beat info","service.name":"packetbeat","system_info":{"beat":{"path":{"config":"C:\\home\\j\\beats\\packetbeat-8.1.3","data":"C:\\home\\j\\beats\\packetbeat-8.1.3\\data","home":"C:\\home\\j\\beats\\packetbeat-8.1.3","logs":"C:\\home\\j\\beats\\packetbeat-8.1.3\\logs"},"type":"packetbeat","uuid":"0b0480b3-ad86-4621-897d-7ba6fc829090"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.308-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1056},"message":"Build info","service.name":"packetbeat","system_info":{"build":{"commit":"271435c21bfd4e2e621d87c04f4b815980626978","libbeat":"8.1.3","time":"2022-04-19T09:56:30.000Z","version":"8.1.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.309-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1059},"message":"Go runtime info","service.name":"packetbeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":1,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.313-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Host info","service.name":"packetbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-17T11:35:37.99-04:00","name":"windcmon","ip":["NOT_WORKING_CLIENT_IP/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17763.2928 (WinBuild.160101.0800)","mac":["08:00:27:2c:bb:c6"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2019 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.2928"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"36e44adc-4866-48d0-a739-96e1f7463d83"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.314-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Process info","service.name":"packetbeat","system_info":{"process":{"cwd":"C:\\home\\j\\beats\\packetbeat-8.1.3","exe":"C:\\home\\j\\beats\\packetbeat-8.1.3\\packetbeat.exe","name":"packetbeat.exe","pid":1436,"ppid":176,"start_time":"2022-05-17T11:52:31.135-0400"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.314-0400","log.origin":{"file.name":"instance/beat.go","file.line":323},"message":"Setup Beat: packetbeat; Version: 8.1.3","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.332-0400","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: windcmon","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.335-0400","log.logger":"npcap","log.origin":{"file.name":"beater/install_npcap.go","file.line":49},"message":"npcap version: Npcap version 1.60, based on libpcap version 1.10.2-PRE-GIT","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.340-0400","log.origin":{"file.name":"procs/procs.go","file.line":103},"message":"Process watcher disabled","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T11:52:34.359-0400","log.logger":"cfgwarn","log.origin":{"file.name":"sip/plugin.go","file.line":67},"message":"BETA: packetbeat SIP protocol is used","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.385-0400","log.origin":{"file.name":"sniffer/device.go","file.line":98},"message":"Resolved device index 0 to device: \\Device\\NPF_{97EE929E-FD64-4EA3-BD41-F84162977019}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.389-0400","log.origin":{"file.name":"instance/beat.go","file.line":489},"message":"packetbeat start running.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.390-0400","log.origin":{"file.name":"procs/procs.go","file.line":103},"message":"Process watcher disabled","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T11:52:34.393-0400","log.logger":"cfgwarn","log.origin":{"file.name":"sip/plugin.go","file.line":67},"message":"BETA: packetbeat SIP protocol is used","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.412-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.418-0400","log.origin":{"file.name":"sniffer/device.go","file.line":98},"message":"Resolved device index 0 to device: \\Device\\NPF_{97EE929E-FD64-4EA3-BD41-F84162977019}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:37.343-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":101},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:53:04.432-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"packetbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":203,"time":{"ms":203}},"total":{"ticks":265,"time":{"ms":265},"value":0},"user":{"ticks":62,"time":{"ms":62}}},"handles":{"open":238},"info":{"ephemeral_id":"39b14446-4196-45e8-811d-9d06b16f9b03","uptime":{"ms":33244},"version":"8.1.3"},"memstats":{"gc_next":13369584,"memory_alloc":7051248,"memory_sys":22667848,"memory_total":17384664,"rss":52977664},"runtime":{"goroutines":54}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":30,"events":{"active":0},"queue":{"max_events":4096}}},"system":{"cpu":{"cores":1}}},"ecs.version":"1.6.0"}}
All these VMs are in the same subnet and windows FW is turned off on the clients.
Thanks!