Osquery manger integration won't work on Windows

Elastic Stack Beats
1

Hi,
I have unsuccessfully been trying to integrate Osquery to elastic-agent on windows but every time I enroll an agent, its status changes to unhealthy a few seconds after enrolling.

image
image2958×898 88.1 KB

image
image2770×469 108 KB

image
image2973×636 66.3 KB

image
image1526×1785 178 KB

image
image1968×377 20.9 KB

The log file is full of entries like this.

{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:24.748Z",
    "log.origin": {
        "file.name": "stateresolver/stateresolver.go",
        "file.line": 48
    },
    "message": "New State ID is USB5jmUA",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:24.748Z",
    "log.origin": {
        "file.name": "stateresolver/stateresolver.go",
        "file.line": 49
    },
    "message": "Converging state requires execution of 4 step(s)",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:24.864Z",
    "log.origin": {
        "file.name": "operation/operator.go",
        "file.line": 284
    },
    "message": "operation 'operation-install' skipped for metricbeat.7.16.3",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:24.864Z",
    "log.origin": {
        "file.name": "operation/operator.go",
        "file.line": 284
    },
    "message": "operation 'operation-start' skipped for metricbeat.7.16.3",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:25.007Z",
    "log.origin": {
        "file.name": "operation/operator.go",
        "file.line": 284
    },
    "message": "operation 'operation-install' skipped for osquerybeat.7.16.3",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:25.075Z",
    "log.origin": {
        "file.name": "log/reporter.go",
        "file.line": 40
    },
    "message": "2022-01-27T00:26:25Z - message: Application: osquerybeat--7.16.3[7711f2db-b675-400f-adae-a952f4690892]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'",
    "ecs.version": "1.6.0"
}
{
    "log.level": "error",
    "@timestamp": "2022-01-27T00:26:25.076Z",
    "log.origin": {
        "file.name": "fleet/fleet_gateway.go",
        "file.line": 180
    },
    "message": "failed to dispatch actions, error: operator: failed to execute step sc-run, error: failed to start 'C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-d420cc\\install\\osquerybeat-7.16.3-windows-x86_64\\osquerybeat': exec: \"C:\\\\Program Files\\\\Elastic\\\\Agent\\\\data\\\\elastic-agent-d420cc\\\\install\\\\osquerybeat-7.16.3-windows-x86_64\\\\osquerybeat\": file does not exist: failed to start 'C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-d420cc\\install\\osquerybeat-7.16.3-windows-x86_64\\osquerybeat': exec: \"C:\\\\Program Files\\\\Elastic\\\\Agent\\\\data\\\\elastic-agent-d420cc\\\\install\\\\osquerybeat-7.16.3-windows-x86_64\\\\osquerybeat\": file does not exist",
    "ecs.version": "1.6.0"
}
{
    "log.level": "error",
    "@timestamp": "2022-01-27T00:26:25.076Z",
    "log.origin": {
        "file.name": "log/reporter.go",
        "file.line": 36
    },
    "message": "2022-01-27T00:26:25Z - message: Application: osquerybeat--7.16.3[7711f2db-b675-400f-adae-a952f4690892]: State changed to FAILED: failed to start 'C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-d420cc\\install\\osquerybeat-7.16.3-windows-x86_64\\osquerybeat': exec: \"C:\\\\Program Files\\\\Elastic\\\\Agent\\\\data\\\\elastic-agent-d420cc\\\\install\\\\osquerybeat-7.16.3-windows-x86_64\\\\osquerybeat\": file does not exist - type: 'ERROR' - sub_type: 'FAILED'",
    "ecs.version": "1.6.0"
}

The logs seem to indicate that a file related to osquerybeat is missing but when checking the download and install directories it seem to be there. Not sure where to check next to get more information. I have uninstalled the agent and reinstalled many times but same error every times.

2

Hi,

could you please open an issue for Beats? It looks like you can easily reproduce it. I'm wondering if you are using any software like antivirus, endpoint security, etc. that may block downloading.

3

Hi @mtojek, So far I have tried on 6 different testing systems: 2x Windows 10 v21H2, 2x Windows 11 and 2x Windows Server 2022. One of the Windows 11 is running Sophos AV but all the other systems are running the Built-in Windows Defender AV.

I get the same error on all of them. Note that I don't get that error the policy only has the system integration. This on happens when I add the Osquery manager integration the policy.

Tested enrolling an ubuntu server and it worked for ubuntu.

4

@mtojek @Bryan_Hamilton
I think it is might be related how the agent child processes (beats) are killed on reboot/uninstall.

It looks like the install directory was possibly left behind, was not able to be deleted on uninstall because it contains only osqueryd binary.
My theory is that:

  1. when uninstalling agent kills osquerybeat
  2. osqueryd.exe is left running
  3. the install directory locked when the file is in use and can not be deleted
  4. next time the agent skips osquerybeat install because the directory is already there

As a workaround can try to either remove the osquerybeat install directory by hand and then start the agent which should reinstall osquerybeat from the downloads.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.