Error Querying Windows System using Osquery Manager on Kibana

Elastic Stack Kibana
1

Hello there,

I have successfully setup and ELK stack v7.17.0 fully integrated with Fleet and Osquery manager.
When I try to query the remote windows agent using osquery live queries, i get the error below;

TypeError: Cannot read properties of undefined (reading 'fields')
    at Object.keepPreviousData [as queryFn] (http://192.168.58.22:5601/46534/bundles/plugin/osquery/8.0.0/osquery.chunk.0.js:3:513413)

What could be causing this issue because i can successfully query Linux based agents with no issues?

Best Regards.

2

Hi @mibeyki , thanks for reporting this :+1:
Unfortunately the bug was not fixed with v7.17, we'll get it fixed with the next version bump: v7.17.1.
Sorry for your inconvenience.

3

Hi @mibeyki, may I know what do you see in the Status tab?

4

Hello @patrykkopycinski, the status tab is shown in the screenshot below.

Selection_747
Selection_7471006×376 27.9 KB

Selection_748
Selection_7481546×405 26 KB

5

Thank you @mibeyki, Sorry for the misleading error showing up, but it seems the issue is with the Windows agent, whenever you see actions undefined it means that the osquerybeat is not up and running properly. Would you mind going to Fleet logs and checking if there is anything in logs or can you verify that on the Windows host you can see elastic agent, osquerybeat and osqueryd are running?

6

Thank you so much @patrykkopycinski for your quick response.

I dont see anything fishy on the Fleet server logs/Elastic agent on both the server and the agent.

Whenever Elastic agent is enrolled, i can only see Elastic Agent service installed.
However, filebeat and metricbeat have scripts for installing individual services;

  • C:\Program Files\Elastic\Agent\data\elastic-agent-93708b\install\filebeat-7.17.0-windows-x86_64\install-service-filebeat.ps1

  • C:\Program Files\Elastic\Agent\data\elastic-agent-93708b\install\metricbeat-7.17.0-windows-x86_64\install-service-metricbeat.ps1

Osquerybeat folder has not such script. However, i can execute the command; osquerbeat.exe -c osquerybeat.yml -e to run it.

And all results in the same error.

7

Could you try to stop the agent and then check if maybe there are any agent-related processes that weren't stopped properly?
I would look for osquerybeat, osqueryd specifically
If so, we think that stopping those processes manually and starting the agent again should help

8

Hello @patrykkopycinski , thank you for your time.

Unfortunately, there is is no service related with osquery;

However, i restarted the Elastic Agent and here is a snippet of the logs as it starts. I can see some osquery processes exiting.

{"log.level":"info","@timestamp":"2022-02-09T08:06:00.658+0300","log.origin":{"file.name":"instance/beat.go","file.line":694},"message":"Beat ID: 0b355099-fa3a-4713-a08a-60550111d2c0","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-02-09T08:06:04.214+0300","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":79},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.582+0300","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":62},"message":"Starting stats endpoint","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.646+0300","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":64},"message":"Metrics endpoint listening on: \\\\.\\pipe\\default-osquerybeat (configured: npipe:///default-osquerybeat)","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.714+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1040},"message":"Beat info","service.name":"osquerybeat","system_info":{"beat":{"path":{"config":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-93708b\\install\\osquerybeat-7.17.0-windows-x86_64","data":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-93708b\\run\\default\\osquerybeat--7.17.0","home":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-93708b\\install\\osquerybeat-7.17.0-windows-x86_64","logs":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-93708b\\install\\osquerybeat-7.17.0-windows-x86_64\\logs"},"type":"osquerybeat","uuid":"0b355099-fa3a-4713-a08a-60550111d2c0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.717+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1049},"message":"Build info","service.name":"osquerybeat","system_info":{"build":{"commit":"93708bd74e909e57ed5d9bea3cf2065f4cc43af3","libbeat":"7.17.0","time":"2022-01-28T09:43:01.000Z","version":"7.17.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.717+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1052},"message":"Go runtime info","service.name":"osquerybeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":2,"version":"go1.17.5"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.727+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1056},"message":"Host info","service.name":"osquerybeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-02-09T07:22:57.5+03:00","name":"WorkBox","ip":["fe80::6826:4fe3:e2b8:d87b/64","169.254.216.123/16","fe80::2055:488e:6da7:aa0d/64","10.0.3.15/24","::1/128","127.0.0.1/8","fe80::5efe:a00:30f/128","2001:0:2851:782c:1c9c:1e69:f5ff:fcf0/64","fe80::1c9c:1e69:f5ff:fcf0/64"],"kernel_version":"10.0.14393.2189 (rs1_release.180329-1711)","mac":["08:00:27:ec:10:78","08:00:27:39:67:b6","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.2189"},"timezone":"EAT","timezone_offset_sec":10800,"id":"e482410a-c215-4628-a12b-18b85954163b"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.727+0300","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1085},"message":"Process info","service.name":"osquerybeat","system_info":{"process":{"cwd":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-93708b\\install\\osquerybeat-7.17.0-windows-x86_64","exe":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-93708b\\install\\osquerybeat-7.17.0-windows-x86_64\\osquerybeat.exe","name":"osquerybeat.exe","pid":960,"ppid":3276,"start_time":"2022-02-09T08:05:55.371+0300"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.727+0300","log.origin":{"file.name":"instance/beat.go","file.line":328},"message":"Setup Beat: osquerybeat; Version: 7.17.0","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.845+0300","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: http://192.168.58.22:9200","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:04.966+0300","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: WorkBox","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:05.033+0300","log.origin":{"file.name":"instance/beat.go","file.line":492},"message":"osquerybeat start running.","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-02-09T08:06:05.035+0300","log.logger":"cfgwarn","log.origin":{"file.name":"management/manager.go","file.line":108},"message":"BETA: Fleet management is enabled","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:05.141+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:05.142+0300","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":109},"message":"Starting fleet management service","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:05.107+0300","log.origin":{"file.name":"service/service_windows.go","file.line":126},"message":"Attempted to register Windows service handlers, but this is not a service. No action necessary","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:05.189+0300","log.logger":"osqueryd_install","log.origin":{"file.name":"beater/install.go","file.line":32},"message":"Check if osqueryd needs to be installed","service.name":"osquerybeat","dir":"C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-93708b\\install\\osquerybeat-7.17.0-windows-x86_64","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:07.399+0300","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":101},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:21.195+0300","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":150},"message":"Status change to Configuring: Updating configuration","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:21.354+0300","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":271},"message":"Applying settings for inputs","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:28.200+0300","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":271},"message":"Applying settings for output","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:29.100+0300","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: http://192.168.58.22:9200","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:29.929+0300","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":213},"message":"retryer: send wait signal to consumer","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:31.534+0300","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":217},"message":"  done","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:31.534+0300","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":219},"message":"retryer: send unwait signal to consumer","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:31.534+0300","log.logger":"publisher","log.origin":{"file.name":"pipeline/retry.go","file.line":223},"message":"  done","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:30.017+0300","log.logger":"osquerybeat","log.origin":{"file.name":"beater/osquery_runner.go","file.line":78},"message":"Start osqueryd","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:34.670+0300","log.logger":"osquerybeat","log.origin":{"file.name":"osqdcli/client.go","file.line":122},"message":"osquery client is connected","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:35.593+0300","log.logger":"osquerybeat","log.origin":{"file.name":"beater/osquerybeat.go","file.line":271},"message":"runOsquery context cancelled, exiting","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:36.907+0300","log.logger":"osquerybeat","log.origin":{"file.name":"beater/osquerybeat.go","file.line":226},"message":"osqueryd process exited","service.name":"osquerybeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-09T08:06:36.925+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":140}},"total":{"ticks":608,"time":{"ms":624},"value":608},"user":{"ticks":468,"time":{"ms":484}}},"handles":{"open":195},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":36274},"version":"7.17.0"},"memstats":{"gc_next":10370720,"memory_alloc":5235528,"memory_sys":18508392,"memory_total":14680800,"rss":41517056},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":4096}}},"system":{"cpu":{"cores":2}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:07:05.195+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":250,"time":{"ms":125}},"total":{"ticks":765,"time":{"ms":156},"value":765},"user":{"ticks":515,"time":{"ms":31}}},"handles":{"open":202},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":65199},"version":"7.17.0"},"memstats":{"gc_next":10370720,"memory_alloc":6901088,"memory_total":16346360,"rss":42364928},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:07:35.207+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":375,"time":{"ms":110}},"total":{"ticks":937,"time":{"ms":188},"value":937},"user":{"ticks":562,"time":{"ms":78}}},"handles":{"open":202},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":95200},"version":"7.17.0"},"memstats":{"gc_next":10370720,"memory_alloc":9177288,"memory_sys":262144,"memory_total":18622560,"rss":42999808},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:08:05.199+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":468,"time":{"ms":93}},"total":{"ticks":1139,"time":{"ms":203},"value":1139},"user":{"ticks":671,"time":{"ms":110}}},"handles":{"open":203},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":125202},"version":"7.17.0"},"memstats":{"gc_next":10206080,"memory_alloc":6648544,"memory_sys":4329680,"memory_total":20903552,"rss":42549248},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:08:35.180+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":593,"time":{"ms":125}},"total":{"ticks":1343,"time":{"ms":172},"value":1343},"user":{"ticks":750,"time":{"ms":47}}},"handles":{"open":203},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":155226},"version":"7.17.0"},"memstats":{"gc_next":10206080,"memory_alloc":8928216,"memory_total":23183224,"rss":43442176},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:09:05.182+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":796,"time":{"ms":203}},"total":{"ticks":1561,"time":{"ms":218},"value":1561},"user":{"ticks":765,"time":{"ms":15}}},"handles":{"open":203},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":185189},"version":"7.17.0"},"memstats":{"gc_next":10120832,"memory_alloc":6495056,"memory_total":25471528,"rss":43327488},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:09:35.188+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":937,"time":{"ms":141}},"total":{"ticks":1749,"time":{"ms":188},"value":1749},"user":{"ticks":812,"time":{"ms":47}}},"handles":{"open":203},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":215190},"version":"7.17.0"},"memstats":{"gc_next":10120832,"memory_alloc":8773496,"memory_total":27749968,"rss":43352064},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:10:05.175+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1109,"time":{"ms":188}},"total":{"ticks":1952,"time":{"ms":219},"value":1952},"user":{"ticks":843,"time":{"ms":31}}},"handles":{"open":203},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":245213},"version":"7.17.0"},"memstats":{"gc_next":10126080,"memory_alloc":6532024,"memory_total":30151720,"rss":43700224},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:10:35.195+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1250,"time":{"ms":125}},"total":{"ticks":2125,"time":{"ms":172},"value":2125},"user":{"ticks":875,"time":{"ms":47}}},"handles":{"open":203},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":275240},"version":"7.17.0"},"memstats":{"gc_next":10126080,"memory_alloc":8693024,"memory_total":32312720,"rss":43749376},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:11:05.164+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1453,"time":{"ms":203}},"total":{"ticks":2390,"time":{"ms":250},"value":2390},"user":{"ticks":937,"time":{"ms":47}}},"handles":{"open":203},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":305196},"version":"7.17.0"},"memstats":{"gc_next":10163632,"memory_alloc":6321216,"memory_total":34592632,"rss":43589632},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-02-09T08:11:35.192+0300","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":184},"message":"Non-zero metrics in the last 30s","service.name":"osquerybeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1625,"time":{"ms":203}},"total":{"ticks":2671,"time":{"ms":312},"value":2671},"user":{"ticks":1046,"time":{"ms":109}}},"handles":{"open":203},"info":{"ephemeral_id":"cc74eb04-a3da-4fef-a52f-0aebf79c809f","uptime":{"ms":335199},"version":"7.17.0"},"memstats":{"gc_next":10163632,"memory_alloc":8604040,"memory_total":36875456,"rss":43610112},"runtime":{"goroutines":33}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}}},"ecs.version":"1.6.0"}}
9

@mibeyki thank you for the logs.
This could potentially happen due to bad osquery configuration.
Could you switch the agent to debug level logging, run the agent and send the logs?
The agent logging level can be changed at the bottom of this page:

Screen Shot 2022-02-09 at 7.27.36 AM
Screen Shot 2022-02-09 at 7.27.36 AM1732×1702 383 KB

Could you also provide the policy configuration, the osquery_manager section? Should look something like this in the policy:

Screen Shot 2022-02-09 at 7.51.42 AM
Screen Shot 2022-02-09 at 7.51.42 AM1304×1476 114 KB

This should give us more information.

10

Thank you very much @aleksmaus. My apologies. I have been afk. I will be sure to share them as soon as i can.
Very many thanks for continued support.

11

Hello @aleksmaus kindly find the requested policy as well as the agent logs after setting the logging level to debug and restarting the agent service.

Default Policy configuration

id: 2016d7cc-135e-5583-9758-3ba01f5a06e5
revision: 4
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://192.168.58.22:9200'
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
    system-1:
      indices:
        - names:
            - logs-system.auth-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-system.syslog-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-system.application-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-system.security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-system.system-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.cpu-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.diskio-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.filesystem-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.fsstat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.load-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.memory-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.network-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.process-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.process.summary-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.socket_summary-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-system.uptime-default
          privileges:
            - auto_configure
            - create_doc
    osquery_manager-1:
      indices:
        - names:
            - logs-osquery_manager.result-default
          privileges:
            - auto_configure
            - create_doc
agent:
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
inputs:
  - id: default-system-policy
    name: system-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 1.6.4
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth-default-system-policy
        data_stream:
          dataset: system.auth
          type: logs
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
      - id: logfile-system.syslog-default-system-policy
        data_stream:
          dataset: system.syslog
          type: logs
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
  - id: default-system-policy
    name: system-1
    revision: 1
    type: winlog
    use_output: default
    meta:
      package:
        name: system
        version: 1.6.4
    data_stream:
      namespace: default
    streams:
      - id: winlog-system.application-default-system-policy
        name: Application
        data_stream:
          dataset: system.application
          type: logs
        condition: '${host.platform} == ''windows'''
        ignore_older: 72h
        tags: null
      - id: winlog-system.security-default-system-policy
        name: Security
        data_stream:
          dataset: system.security
          type: logs
        condition: '${host.platform} == ''windows'''
        tags: null
      - id: winlog-system.system-default-system-policy
        name: System
        data_stream:
          dataset: system.system
          type: logs
        condition: '${host.platform} == ''windows'''
        tags: null
  - id: default-system-policy
    name: system-1
    revision: 1
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 1.6.4
    data_stream:
      namespace: default
    streams:
      - id: system/metrics-system.cpu-default-system-policy
        data_stream:
          dataset: system.cpu
          type: metrics
        metricsets:
          - cpu
        cpu.metrics:
          - percentages
          - normalized_percentages
        period: 10s
      - id: system/metrics-system.diskio-default-system-policy
        data_stream:
          dataset: system.diskio
          type: metrics
        metricsets:
          - diskio
        diskio.include_devices: null
        period: 10s
      - id: system/metrics-system.filesystem-default-system-policy
        data_stream:
          dataset: system.filesystem
          type: metrics
        metricsets:
          - filesystem
        period: 1m
        processors:
          - drop_event.when.regexp:
              system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.fsstat-default-system-policy
        data_stream:
          dataset: system.fsstat
          type: metrics
        metricsets:
          - fsstat
        period: 1m
        processors:
          - drop_event.when.regexp:
              system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.load-default-system-policy
        data_stream:
          dataset: system.load
          type: metrics
        metricsets:
          - load
        condition: '${host.platform} != ''windows'''
        period: 10s
      - id: system/metrics-system.memory-default-system-policy
        data_stream:
          dataset: system.memory
          type: metrics
        metricsets:
          - memory
        period: 10s
      - id: system/metrics-system.network-default-system-policy
        data_stream:
          dataset: system.network
          type: metrics
        metricsets:
          - network
        period: 10s
        network.interfaces: null
      - id: system/metrics-system.process-default-system-policy
        data_stream:
          dataset: system.process
          type: metrics
        metricsets:
          - process
        period: 10s
        process.include_top_n.by_cpu: 5
        process.include_top_n.by_memory: 5
        process.cmdline.cache.enabled: true
        process.cgroups.enabled: false
        process.include_cpu_ticks: false
        processes:
          - .*
      - id: system/metrics-system.process.summary-default-system-policy
        data_stream:
          dataset: system.process.summary
          type: metrics
        metricsets:
          - process_summary
        period: 10s
      - id: system/metrics-system.socket_summary-default-system-policy
        data_stream:
          dataset: system.socket_summary
          type: metrics
        metricsets:
          - socket_summary
        period: 10s
      - id: system/metrics-system.uptime-default-system-policy
        data_stream:
          dataset: system.uptime
          type: metrics
        metricsets:
          - uptime
        period: 10s
  - id: da30823c-2ed3-493e-9531-10640caab071
    name: osquery_manager-1
    revision: 1
    type: osquery
    use_output: default
    meta:
      package:
        name: osquery_manager
        version: 1.0.0
    data_stream:
      namespace: default
fleet:
  hosts:
    - 'http://192.168.58.22:8220'
12

HOLD ON! It seems it is working fine now! Any changes in the code?

I just ran the query so as to get the agent logs and boom, RESULTS.

Selection_792
Selection_7921468×598 94.4 KB

13

I will roll out on other nodes and provide feedback on how it goes. many thanks

14

My theory at the moment: this seems most likely could happen when the agent is upgraded and upgrades the beats. If any of the beat child processes are left running it could prevent the beat install directory from being properly removed leaving corrupted beat install directory.
Fresh reinstall should work as a workaround at the moment.

We added some more changes that should help with this problem especially on windows, some changes will make into upcoming 7.17.1, some into 8.x versions, specifically:

  • The child processes tree better handling on agent shutdown or when it gets killed. all the child processes should be killed too now.
  • Install verification for osquerybeat, allows to detect the corrupted osquerybeat installs better and reinstall osquerybeat seamlessly for the user.
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.