Packetbeat not sending to Logstash after Elastic-Agent uninstall

While experimenting with elastic-agent/fleet-server I noticed that once the agent was uninstalled, packetbeat didn't seem able to send to logstash anymore. No more indexes, nothing.

The sequence is basically:

  • Packetbeat ok and sending to logstash (custom indexes created and visible in kibana)
  • Stop packetbeat (i run it manually while testing), install/enroll elastic-agent and test (elastic-agent works as intended)
  • Unenroll/uninstall elastic-agent
  • Run packetbeat again with zero data sent (no backoffice connection in output)

Nothing was changed in Packetbeat's config.

I've had this happen while connecting a server and it's agent/beats to a fleet-enabled ELK stack, as well as connecting a formerly agent-enabled server to a new non-fleet-enabled ELK stack (classic beats only).

Winlogbeat does not seem affected.

I realized that i'd forgotten to mention versions -- i've observed this behavior with with versions above 8.0.0 as i'd not used elastic-agent before that version. I am currently on 8.1.0.

The rumors of my giving up this post were highly overrated. :slight_smile:

Made a few changes to my setup and managed to once again reproduce the problem. Formerly i'd done this with manually run binaries from downloaded archives. Then i moved on to a docker setup (full ELK stack in docker with fleet server running on the host OS). And now this...

My new setup is as follows:

  • Dedicated ubuntu server
  • Elasticsearch, Kibana, Logstash, Fleet Server in v.8.1.3 (the version is purposeful as i want to test an upgrade to 8.2+) and installed from official DEB packages.
  • Single-node, nothing fancy
  • No agents connected to the fleet server as of yet, only beats (specifically Packetbeat and Winlogbeat)

I still haven't seen any direct evidence of a connection error in the logs even with debug mode activated, but that may be due to my lack of experience with the product. So here is the output from a working packetbeat connection:

{"log.level":"info","@timestamp":"2022-05-17T12:47:10.839-0400","log.origin":{"file.name":"instance/beat.go","file.line":669},"message":"Home path: [C:\\home\\j\\beats\\packetbeat-8.1.3] Config path: [C:\\home\\j\\beats\\packetbeat-8.1.3] Data path: [C:\\home\\j\\beats\\packetbeat-8.1.3\\data] Logs path: [C:\\home\\j\\beats\\packetbeat-8.1.3\\logs]","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:10.846-0400","log.origin":{"file.name":"instance/beat.go","file.line":677},"message":"Beat ID: abeb4cc3-a5b5-4665-9625-973990d2d259","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T12:47:13.858-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.877-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1047},"message":"Beat info","service.name":"packetbeat","system_info":{"beat":{"path":{"config":"C:\\home\\j\\beats\\packetbeat-8.1.3","data":"C:\\home\\j\\beats\\packetbeat-8.1.3\\data","home":"C:\\home\\j\\beats\\packetbeat-8.1.3","logs":"C:\\home\\j\\beats\\packetbeat-8.1.3\\logs"},"type":"packetbeat","uuid":"abeb4cc3-a5b5-4665-9625-973990d2d259"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.882-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1056},"message":"Build info","service.name":"packetbeat","system_info":{"build":{"commit":"271435c21bfd4e2e621d87c04f4b815980626978","libbeat":"8.1.3","time":"2022-04-19T09:56:30.000Z","version":"8.1.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.882-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1059},"message":"Go runtime info","service.name":"packetbeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":1,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.885-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Host info","service.name":"packetbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-17T11:56:29.8-04:00","name":"win10client","ip":["fe80::9cf6:869c:8fb2:709b/64","WORKING_CLIENT_IP/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.19041.1706 (WinBuild.160101.0800)","mac":["08:00:27:ed:6f:6a"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"19044.1706"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"ce58b856-23f4-4c56-8def-ee43ed3ae2d3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.886-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Process info","service.name":"packetbeat","system_info":{"process":{"cwd":"C:\\home\\j\\beats\\packetbeat-8.1.3","exe":"C:\\home\\j\\beats\\packetbeat-8.1.3\\packetbeat.exe","name":"packetbeat.exe","pid":2360,"ppid":5016,"start_time":"2022-05-17T12:47:10.690-0400"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.886-0400","log.origin":{"file.name":"instance/beat.go","file.line":323},"message":"Setup Beat: packetbeat; Version: 8.1.3","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.900-0400","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: win10client","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.903-0400","log.logger":"npcap","log.origin":{"file.name":"beater/install_npcap.go","file.line":49},"message":"npcap version: Npcap version 1.60, based on libpcap version 1.10.2-PRE-GIT","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.908-0400","log.origin":{"file.name":"procs/procs.go","file.line":103},"message":"Process watcher disabled","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T12:47:13.927-0400","log.logger":"cfgwarn","log.origin":{"file.name":"sip/plugin.go","file.line":67},"message":"BETA: packetbeat SIP protocol is used","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.944-0400","log.origin":{"file.name":"sniffer/device.go","file.line":98},"message":"Resolved device index 0 to device: \\Device\\NPF_{14FB7DA4-DA62-4E32-BFF4-92C5A25C40D1}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.948-0400","log.origin":{"file.name":"instance/beat.go","file.line":489},"message":"packetbeat start running.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.949-0400","log.origin":{"file.name":"procs/procs.go","file.line":103},"message":"Process watcher disabled","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T12:47:13.951-0400","log.logger":"cfgwarn","log.origin":{"file.name":"sip/plugin.go","file.line":67},"message":"BETA: packetbeat SIP protocol is used","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.964-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:13.969-0400","log.origin":{"file.name":"sniffer/device.go","file.line":98},"message":"Resolved device index 0 to device: \\Device\\NPF_{14FB7DA4-DA62-4E32-BFF4-92C5A25C40D1}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:16.882-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":101},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:31.023-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(async(tcp://MY_SERVER_IP:5044))","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:31.027-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(async(tcp://MY_SERVER_IP:5044)) established","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:43.984-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"packetbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":484,"time":{"ms":484}},"total":{"ticks":499,"time":{"ms":499},"value":499},"user":{"ticks":15,"time":{"ms":15}}},"handles":{"open":189},"info":{"ephemeral_id":"59cb2633-8bdc-4c88-8a1d-236fa1d8e853","uptime":{"ms":33205},"version":"8.1.3"},"memstats":{"gc_next":11707680,"memory_alloc":6820360,"memory_sys":22929992,"memory_total":23320688,"rss":52240384},"runtime":{"goroutines":55}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":23,"active":0,"batches":4,"total":23},"read":{"bytes":24},"type":"logstash","write":{"bytes":12808}},"pipeline":{"clients":30,"events":{"active":0,"published":23,"retry":8,"total":23},"queue":{"acked":23,"max_events":4096}}},"system":{"cpu":{"cores":1}}},"ecs.version":"1.6.0"}}

Please notice the two lines where i replaced my server ip with "MY_SERVER_IP".

{"log.level":"info","@timestamp":"2022-05-17T12:47:31.023-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(async(tcp://MY_SERVER_IP:5044))","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T12:47:31.027-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(async(tcp://MY_SERVER_IP:5044)) established","service.name":"packetbeat","ecs.version":"1.6.0"}

Aside from the obvious -- i.e. a brand new index in my kibana -- it's my indication that a successful connection was established.

Now here's the same output from a different client -- no connection even though the day before it did and generated a new index.

{"log.level":"info","@timestamp":"2022-05-17T11:52:31.292-0400","log.origin":{"file.name":"instance/beat.go","file.line":669},"message":"Home path: [C:\\home\\j\\beats\\packetbeat-8.1.3] Config path: [C:\\home\\j\\beats\\packetbeat-8.1.3] Data path: [C:\\home\\j\\beats\\packetbeat-8.1.3\\data] Logs path: [C:\\home\\j\\beats\\packetbeat-8.1.3\\logs]","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:31.293-0400","log.origin":{"file.name":"instance/beat.go","file.line":677},"message":"Beat ID: 0b0480b3-ad86-4621-897d-7ba6fc829090","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T11:52:34.303-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.308-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1047},"message":"Beat info","service.name":"packetbeat","system_info":{"beat":{"path":{"config":"C:\\home\\j\\beats\\packetbeat-8.1.3","data":"C:\\home\\j\\beats\\packetbeat-8.1.3\\data","home":"C:\\home\\j\\beats\\packetbeat-8.1.3","logs":"C:\\home\\j\\beats\\packetbeat-8.1.3\\logs"},"type":"packetbeat","uuid":"0b0480b3-ad86-4621-897d-7ba6fc829090"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.308-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1056},"message":"Build info","service.name":"packetbeat","system_info":{"build":{"commit":"271435c21bfd4e2e621d87c04f4b815980626978","libbeat":"8.1.3","time":"2022-04-19T09:56:30.000Z","version":"8.1.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.309-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1059},"message":"Go runtime info","service.name":"packetbeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":1,"version":"go1.17.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.313-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Host info","service.name":"packetbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-05-17T11:35:37.99-04:00","name":"windcmon","ip":["NOT_WORKING_CLIENT_IP/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17763.2928 (WinBuild.160101.0800)","mac":["08:00:27:2c:bb:c6"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2019 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.2928"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"36e44adc-4866-48d0-a739-96e1f7463d83"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.314-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Process info","service.name":"packetbeat","system_info":{"process":{"cwd":"C:\\home\\j\\beats\\packetbeat-8.1.3","exe":"C:\\home\\j\\beats\\packetbeat-8.1.3\\packetbeat.exe","name":"packetbeat.exe","pid":1436,"ppid":176,"start_time":"2022-05-17T11:52:31.135-0400"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.314-0400","log.origin":{"file.name":"instance/beat.go","file.line":323},"message":"Setup Beat: packetbeat; Version: 8.1.3","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.332-0400","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: windcmon","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.335-0400","log.logger":"npcap","log.origin":{"file.name":"beater/install_npcap.go","file.line":49},"message":"npcap version: Npcap version 1.60, based on libpcap version 1.10.2-PRE-GIT","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.340-0400","log.origin":{"file.name":"procs/procs.go","file.line":103},"message":"Process watcher disabled","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T11:52:34.359-0400","log.logger":"cfgwarn","log.origin":{"file.name":"sip/plugin.go","file.line":67},"message":"BETA: packetbeat SIP protocol is used","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.385-0400","log.origin":{"file.name":"sniffer/device.go","file.line":98},"message":"Resolved device index 0 to device: \\Device\\NPF_{97EE929E-FD64-4EA3-BD41-F84162977019}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.389-0400","log.origin":{"file.name":"instance/beat.go","file.line":489},"message":"packetbeat start running.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.390-0400","log.origin":{"file.name":"procs/procs.go","file.line":103},"message":"Process watcher disabled","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-17T11:52:34.393-0400","log.logger":"cfgwarn","log.origin":{"file.name":"sip/plugin.go","file.line":67},"message":"BETA: packetbeat SIP protocol is used","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.412-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:34.418-0400","log.origin":{"file.name":"sniffer/device.go","file.line":98},"message":"Resolved device index 0 to device: \\Device\\NPF_{97EE929E-FD64-4EA3-BD41-F84162977019}","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:52:37.343-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":101},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"packetbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-17T11:53:04.432-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"packetbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":203,"time":{"ms":203}},"total":{"ticks":265,"time":{"ms":265},"value":0},"user":{"ticks":62,"time":{"ms":62}}},"handles":{"open":238},"info":{"ephemeral_id":"39b14446-4196-45e8-811d-9d06b16f9b03","uptime":{"ms":33244},"version":"8.1.3"},"memstats":{"gc_next":13369584,"memory_alloc":7051248,"memory_sys":22667848,"memory_total":17384664,"rss":52977664},"runtime":{"goroutines":54}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":30,"events":{"active":0},"queue":{"max_events":4096}}},"system":{"cpu":{"cores":1}}},"ecs.version":"1.6.0"}}

All these VMs are in the same subnet and windows FW is turned off on the clients.

Thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.