Osquery results don't come in: "matching app is not found for action input: osquery"

Elastic Security
,
1

Hey,

unfortunately, I managed to break my OSQuery integrations again.

A simple query like SELECT * FROM os_version; yields the error message matching app is not found for action input: osquery for most of my hosts.
Edit: The error message shows up in the "error" column of the query results in Kibana.
I did some trial and error, and it seems that the error shows up as soon as the endpoint integration is added to the host:

I have two policies that are identical except for the endpoint integration. When I assign the one with endpoint integration to the host, I get the error message. As soon as I remove the endpoint integration, osquery results work again for this host.

This is the case across 7.15 and 7.16.
The cluster is on 7.16.1 (incl. fleet server). Integration versions involved:

  • Endpoint 1.2.2
  • OSquery manager 0.8.0

The logs in C:\Program Files\Elastic\Agent\data\elastic-agent-4bcd95\logs\default\osquerybeat-json.log don't show anything related to queries, just start/stop of the service and connecting to Elasticsearch.

Any tips on where to search further?

Thanks!

2

The " matching app is not found for action input: osquery" errors are reported if there is no osquerybeat active/running on the agent for any reason. So the agent doesn't have a handler app for the specified type of action: osquerybeat not running -> nothing can handler osquery input type -> logs error

It's strange that the adding and removing the endpoint integration breaks the osquery. Seems like it could be some issue with saving the policy correctly on kibana side.

I would check the raw policy before and after you add endpoint integration, possibly reload UI before that in order to avoid any possible caching on the client.
You should be able to see osquery type in the policy if everything is saved correctly. If it's missing after you added the endpoint (due to the error that you mentioned when adding it), then't is possibly a bug on kibana side when adding the endpoint integration code.

Screen Shot 2021-12-20 at 1.07.51 PM
Screen Shot 2021-12-20 at 1.07.51 PM1920×1754 118 KB

1 Like
3

Hey,

thanks alot for the hint. Indeed, the actual policy was out of sync with the GUI and was missing the OSQuery Integration.

I managed to fix this by stopping to mess with endpoint protection and instead removing and re-adding the osquery manager integration.

I have a hunch that this situation came to be because of the new "upgrade all integrations" feature, which I used to upgrade all osquery manager integrations in all policies to 0.8.0.
This operation failed, because all policies had the same name ("osquery_manager-1"). Maybe during this, the integration was removed from the policy, but kibana thought it was still there.

4

@nemhods

Thanks for following back up and letting us know the result.

For clarification on our end, when you removed and re-added the OSQuery integration, did it start to work again with Endpoint Security on the same Policy? Or are you still encountering the problem where you cannot have both OSQuery and Endpoint Security on the same Policy?

5

Hey, no, endpoint security seems to be fine.
When I was troubleshooting, I had two policies which i assumed to be the same except for the endpoint integration, thus I though this was the cause.
The true difference between the two policies was that one still had the osquery manager integration intact.

Come to think of it: I think the policy that still worked was the only policy that also had a successful mass-upgrade of the osquery manager integration. I can't prove it, but I suspect that the upgrades failing halfway through might have been the culprit for all the other policies.

6

Hey @nemhods, it seems that currently, we are having issues with upgrading osquery_manager integration. From my small investigation, I have found out that after the migration, the agent policy doesn't contain an osquery input anymore which is necessary to be able to operate osquery properly.
As a workaround, I can offer for now to remove and add osquery_manager integration to the affected policies (if you had any packs applied to those policies they need to be re-applied as well)
Please let me know if that helped

2 Likes
7

Hey,
that makes sense. Thanks for the heads up! I have already re-added all osquery_manager integrations.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.