No Result from osquery elastic agent 7.14.0

Elastic Stack Beats
,
1

I have tried to run the query using osquery manager on one of my agent
select * from users

However, no result is returned.

I got the below message from my agent logs

{"log.level":"warn","@timestamp":"2021-08-11T08:27:15.545-0700","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":405},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc03d18f42d087974, ext:1437107101, loc:(*time.Location)(0x3ea4d40)}, Meta:{"raw_index":"logs-osquery_manager.result-default"}, Fields:{"action_data":{"id":"ca13e0cb-4755-410d-aa6f-49f4422ca950","query":"Select * from users"},"action_id":"51df55bd-b240-4bad-93cb-0b01f9126d4b","agent":{"ephemeral_id":"74eec063-134c-4ad5-9828-e5cd62d5c667","hostname":"RCA","id":"27e57477-317f-4e78-bb4f-ae67e4c83557","name":"RCA","type":"osquerybeat","version":"7.14.0"},"ecs":{"version":"1.10.0"},"elastic_agent":{"id":"27e57477-317f-4e78-bb4f-ae67e4c83557","snapshot":false,"version":"7.14.0"},"host":{"architecture":"x86_64","hostname":"RCA","id":"ac62cad1-78ec-47da-95e4-db96fecaa4f9","ip":["fe80::9097:6876:f9c4:c509","192.168.86.170"],"mac":["00:15:5d:0f:02:54"],"os":{"build":"17763.2061","family":"windows","kernel":"10.0.17763.2061 (WinBuild.160101.0800)","name":"Windows Server 2019 Standard Evaluation","platform":"windows","type":"windows","version":"10.0"}},"osquery":{"description":"Built-in account for administering the computer/domain","directory":"C:\\Users\\Administrator","gid":1000,"gid_signed":1000,"shell":"C:\\Windows\\system32\\cmd.exe","type":"local","uid":500,"uid_signed":500,"username":"Administrator","uuid":"S-1-5-21-3014340581-434985446-458634352-500"},"type":"RCA"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [TfjCNXsBZgKK5Hs7miay] of user [elastic/fleet-server] on indices [logs-osquery_manager.result-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}","service.name":"osquerybeat","event.dataset":"osquerybeat-json.log","ecs.version":"1.6.0"}

Any ideas how to resolve that?

2

According to this log the osquery results datastream can't be created due to permissions set for that API key. The datastream is created upon the first results document sent to Elasticsearch.

The permissions for the API key are derived from the policy.

Could you please check the output_permissions in the agent policy it should have a blob that looks something like this?

output_permissions:
  default:
...............
    osquery_manager:
      indices:
        - names:
            - logs-osquery_manager.result-default
          privileges:
            - auto_configure
            - create_doc

Is this a clean 7.14 install or was it upgraded from 7.13?
There was the integrations permissions tightening done in 7.14.

Was the policy revision with the right permissions applied to the agent?
If you could provide your policy configuration (minus any sensitive info/creds), that would be helpful.

If the permissions look correct, one more thing you could try is to reenroll the agent. This will force to issue the new API key.

One last thing as a workaround for this problem is to create the data stream from kibana dev tools.

PUT /_data_stream/logs-osquery_manager.result-default

Once the data stream is created the osquerybeat should not have problems sending up the results.
It still would be helpful for us to figure out the original cause of the problem.

1 Like
3

Hi,
Thanks for the reply.
Please find below my policy configuration

id: 3666a000-f6de-11eb-9734-0bd4bc254b66
revision: 15
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://test.com:9200'
output_permissions:
  default:
    windows-1:
      indices:
        - names:
            - logs-windows.sysmon_operational-default
          privileges:
            - auto_configure
            - create_doc
    osquery_manager-1:
      indices:
        - names:
            - logs-osquery_manager.result-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
agent:
  monitoring:
    enabled: false
    logs: false
    metrics: false
inputs:
  - id: 20d16faf-5b21-4313-8ec6-8117dc5637fc
    name: fleet_server-1
    revision: 1
    type: fleet-server
    use_output: default
    meta:
      package:
        name: fleet_server
        version: 1.0.0
    data_stream:
      namespace: default
    server:
      port: 8220
      host: 0.0.0.0
  - id: 48b22533-d97e-4ed6-a625-4eb38e56c5d7
    name: windows-1
    revision: 2
    type: winlog
    use_output: default
    meta:
      package:
        name: windows
        version: 1.0.0
    data_stream:
      namespace: default
    streams:
      - id: winlog-windows.sysmon_operational-48b22533-d97e-4ed6-a625-4eb38e56c5d7
        name: Microsoft-Windows-Sysmon/Operational
        data_stream:
          dataset: windows.sysmon_operational
          type: logs
        condition: '${host.platform} == ''windows'''
        tags:
          - default_sysmon_tag
  - id: 1476bace-0d75-40d5-b628-3106de362628
    name: osquery_manager-1
    revision: 1
    type: osquery
    use_output: default
    meta:
      package:
        name: osquery_manager
        version: 0.3.2
    data_stream:
      namespace: default
fleet:
  hosts:
    - 'https://test.com:8220'

I have re-enrolled the agent and also create the data stream from kibana dev and now face the new error:

Error: EsError
    at search_interceptor_SearchInterceptor.handleSearchError (https://test.com/42747/bundles/plugin/data/kibana/data.plugin.js:1:414849)
    at t.selector (https://test.com/42747/bundles/plugin/data/kibana/data.plugin.js:1:418880)
    at t.error (https://test.com/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:329:85314)
    at t._error (https://test.com/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:53852)
    at t.error (https://test.com/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:53546)
    at t.error (https://test.com/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:298:51934)
    at Object.error (https://test.com/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:329:117577)
    at t.__tryOrUnsub (https://test.com/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:55574)
    at t.error (https://test.com/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:54975)
    at t._error (https://test.com/42747/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:53852)
4

Hi,

I have managed to get the query results.

It seems that for the osquery manager integration, the default namespace cannot be set to other namespace other than default.

5

I can confirm, had the same issue, and setting the Integration namespace to "default" solved the problem.

6

Hi @cheapsupps - I'm the PM for the team who's been working on the Osquery Manager integration. Thank you for raising this issue. You are correct that at the moment, the integration namespace must be set to default. We are going to look into how to support non-default namespaces in the future, but for now, in an upcoming release we will make this requirement clearer when setting up the integration to help avoid these problems.

1 Like
7

This resolved my issue as well, thanks! I had changed the default namespace to an org name which helps us organize the data more efficiently. It'd be great to support additional namespaces for osquery in the future.

8

hi, so this issue will be fixed in release 7.14.1? thanks

1 Like
9

No, unfortunately we won't have a fix for this in 7.14.1. For now, to use Osquery Manager, we recommend using the default namespace. There are still some technical issues that we're looking into to support non-default namespaces for this integration.

10

Hi, for air-gap environment, I notice that the elastic-agent is not downloading the package from private epr registry, it actually need to download the package from internet. Is there any plan to solve this issue?

11

Do you have the Enterprise license for the air-gap environments ?

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

13

Hi @cheapsupps. Starting in 7.16, osquerybeat is now bundled with Elastic Agent, so the Osquery Manager setup should work better now for an air-gap environment.

14

An update on this: as of 7.16, Osquery Manager supports custom integration namespaces, so this should no longer be an issue.

1 Like