Osquery manger integration won't work on Windows

[Bryan_Hamilton]

Hi,
I have unsuccessfully been trying to integrate Osquery to elastic-agent on windows but every time I enroll an agent, its status changes to unhealthy a few seconds after enrolling.

The log file is full of entries like this.

{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:24.748Z",
    "log.origin": {
        "file.name": "stateresolver/stateresolver.go",
        "file.line": 48
    },
    "message": "New State ID is USB5jmUA",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:24.748Z",
    "log.origin": {
        "file.name": "stateresolver/stateresolver.go",
        "file.line": 49
    },
    "message": "Converging state requires execution of 4 step(s)",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:24.864Z",
    "log.origin": {
        "file.name": "operation/operator.go",
        "file.line": 284
    },
    "message": "operation 'operation-install' skipped for metricbeat.7.16.3",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:24.864Z",
    "log.origin": {
        "file.name": "operation/operator.go",
        "file.line": 284
    },
    "message": "operation 'operation-start' skipped for metricbeat.7.16.3",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:25.007Z",
    "log.origin": {
        "file.name": "operation/operator.go",
        "file.line": 284
    },
    "message": "operation 'operation-install' skipped for osquerybeat.7.16.3",
    "ecs.version": "1.6.0"
}
{
    "log.level": "info",
    "@timestamp": "2022-01-27T00:26:25.075Z",
    "log.origin": {
        "file.name": "log/reporter.go",
        "file.line": 40
    },
    "message": "2022-01-27T00:26:25Z - message: Application: osquerybeat--7.16.3[7711f2db-b675-400f-adae-a952f4690892]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'",
    "ecs.version": "1.6.0"
}
{
    "log.level": "error",
    "@timestamp": "2022-01-27T00:26:25.076Z",
    "log.origin": {
        "file.name": "fleet/fleet_gateway.go",
        "file.line": 180
    },
    "message": "failed to dispatch actions, error: operator: failed to execute step sc-run, error: failed to start 'C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-d420cc\\install\\osquerybeat-7.16.3-windows-x86_64\\osquerybeat': exec: \"C:\\\\Program Files\\\\Elastic\\\\Agent\\\\data\\\\elastic-agent-d420cc\\\\install\\\\osquerybeat-7.16.3-windows-x86_64\\\\osquerybeat\": file does not exist: failed to start 'C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-d420cc\\install\\osquerybeat-7.16.3-windows-x86_64\\osquerybeat': exec: \"C:\\\\Program Files\\\\Elastic\\\\Agent\\\\data\\\\elastic-agent-d420cc\\\\install\\\\osquerybeat-7.16.3-windows-x86_64\\\\osquerybeat\": file does not exist",
    "ecs.version": "1.6.0"
}
{
    "log.level": "error",
    "@timestamp": "2022-01-27T00:26:25.076Z",
    "log.origin": {
        "file.name": "log/reporter.go",
        "file.line": 36
    },
    "message": "2022-01-27T00:26:25Z - message: Application: osquerybeat--7.16.3[7711f2db-b675-400f-adae-a952f4690892]: State changed to FAILED: failed to start 'C:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-d420cc\\install\\osquerybeat-7.16.3-windows-x86_64\\osquerybeat': exec: \"C:\\\\Program Files\\\\Elastic\\\\Agent\\\\data\\\\elastic-agent-d420cc\\\\install\\\\osquerybeat-7.16.3-windows-x86_64\\\\osquerybeat\": file does not exist - type: 'ERROR' - sub_type: 'FAILED'",
    "ecs.version": "1.6.0"
}

The logs seem to indicate that a file related to osquerybeat is missing but when checking the download and install directories it seem to be there. Not sure where to check next to get more information. I have uninstalled the agent and reinstalled many times but same error every times.

[mtojek (Marcin Tojek)]

Hi,

could you please open an issue for Beats? It looks like you can easily reproduce it. I'm wondering if you are using any software like antivirus, endpoint security, etc. that may block downloading.

[Bryan_Hamilton]

Hi @mtojek, So far I have tried on 6 different testing systems: 2x Windows 10 v21H2, 2x Windows 11 and 2x Windows Server 2022. One of the Windows 11 is running Sophos AV but all the other systems are running the Built-in Windows Defender AV.

I get the same error on all of them. Note that I don't get that error the policy only has the system integration. This on happens when I add the Osquery manager integration the policy.

Tested enrolling an ubuntu server and it worked for ubuntu.

[aleksmaus (Aleksandr Maus)]

@mtojek @Bryan_Hamilton
I think it is might be related how the agent child processes (beats) are killed on reboot/uninstall.

It looks like the install directory was possibly left behind, was not able to be deleted on uninstall because it contains only osqueryd binary.
My theory is that:

1. when uninstalling agent kills osquerybeat
2. osqueryd.exe is left running
3. the install directory locked when the file is in use and can not be deleted
4. next time the agent skips osquerybeat install because the directory is already there

As a workaround can try to either remove the osquerybeat install directory by hand and then start the agent which should reinstall osquerybeat from the downloads.