How to query list of offline agents using the Fleet API?

Elastic Stack Kibana
1

I looked at these two docs:

and came up with this query:

curl --request GET --url 'https://mykibana/api/fleet/agents?kuery=status:offline' \
       --header 'Accept: */*' \
       --header 'Authorization: ApiKey mykey' \
       --header 'Cache-Control: no-cache'  \
       --header 'Connection: keep-alive' \
       --header 'Content-Type: application/json' \
      --header 'kbn-xsrf: xx

but that returned:

{
  "list": [],
  "items": [],
  "total": 0,
  "totalInactive": 0,
  "page": 1,
  "perPage": 20
}

What is the correct query for listing all offline agents?

Also can someone tell me how to get the Elasticsearch schema
for Fleet agents, so that I can do arbitrary queries via the kuery parameter?

2

I notice in the code for fleet:

that certain queries are being constructed:

status:unenrolled
status.online
(status:error or status:degraded)
(status:updating or status:unenrolling or status:enrolling)
status:inactive

I don't understand how to pass those queries to the kuery parameter in the Fleet API

3

Hey Craig,

The api/fleet/agents?kuery=status:offline query should work fine.

Could you share the agent statuses in the output of api/fleet/agents?

4

If I just do:

https://mykibana/api/fleet/agents

I get hundreds of agents (since I am running hundreds). A bit too big to post here.

One of the offline agents looks like this:

{
  "list": [
    {
      "id": "xxx",
      "access_api_key_id": "xxx",
      "action_seq_no": [
        -1
      ],
      "active": true,
      "agent": {
        "id": "xxx",
        "version": "8.4.2"
      },
      "enrolled_at": "2023-08-16T05:52:59Z",
      "local_metadata": {
        "elastic": {
          "agent": {
            "build.original": "8.4.2 (build: d3eb3e8a02d96bf48d11145fb089e118f75bd202 at 2022-09-14 22:33:40 +0000 UTC)",
            "id": "xxx,
            "log_level": "info",
            "snapshot": false,
            "upgradeable": false,
            "version": "8.4.2"
          }
        },
        "host": {
          "architecture": "x86_64",
          "hostname": "xxx",
          "id": "xxx",
          "ip": [
            "xxx"
          ],
          "mac": [
            "xxx"
          ],
          "name": "xxx"
        },
        "os": {
          "family": "debian",
          "full": "Ubuntu focal(20.04.5 LTS (Focal Fossa))",
          "kernel": "5.4.209-116.367.amzn2.x86_64",
          "name": "Ubuntu",
          "platform": "ubuntu",
          "version": "20.04.5 LTS (Focal Fossa)"
        }
      },
      "policy_id": "8fab94c0-69e4-11ed-be84-13a1190e0b89",
      "type": "PERMANENT",
      "default_api_key": "xxx",
      "default_api_key_id": "xxx",
      "policy_output_permissions_hash": "dc9401abb0dfda2c534789e7b1758981eb78621b4ea9c548b40cdb3c42d351a0",
      "policy_revision_idx": 12,
      "policy_coordinator_idx": 1,
      "updated_at": "2023-08-16T06:19:14Z",
      "last_checkin_status": "degraded",
      "last_checkin": "2023-08-16T06:19:06Z",
      "policy_revision": 12,
      "status": "offline",
      "packages": [],
      "sort": [
        1692165179000,
        "xxx"
      ]
    }
  ]
}

If I do:

https://mykibana/api/fleet/agents?kuery=status:offline

I get:

{"list":[],"items":[],"total":0,"totalInactive":0,"page":1,"perPage":20}
5

Quick check: does the query work for other status types, e.g. status:online?

6

I checked for other query times such as

kuery=status:online
kuery=status:degraded

and those queries did not work either

I don't know if it makes a difference but I am using Elastic 8.4.2

7

What's the output of /api/fleet/agent_status?

8

If I access /api/fleet/agent_status

I get:

{
  "results": {
    "total": 207,
    "inactive": 11861,
    "online": 92,
    "error": 9,
    "offline": 106,
    "updating": 0,
    "other": 11861,
    "events": 0
  }
}
9

Could you try grabbing the query from the browser's dev tools Network tab when you load the Fleet page in the UI? You can filter by agent status, although that's probably beside the point at this stage. The query will be more complex than what you need, but it would be good to validate whether that works with curl.

10

On my Elastic 8.4.2 system, in the Fleet UI, if I try to select offline agents, I get a browser query (which I found in the network console) that looks like:

kbn:/api/fleet/agents?page=1&perPage=20&kuery=last_checkin%20%3C%20now-300s%20AND%20not%20((last_checkin_status%3Aerror%20or%20last_checkin_status%3Adegraded)%20AND%20not%20(((upgrade_started_at%3A*)%20and%20not%20(upgraded_at%3A*))%20or%20(not%20(last_checkin%3A*))%20or%20(unenrollment_started_at%3A*)))%20AND%20not%20(%20((upgrade_started_at%3A*)%20and%20not%20(upgraded_at%3A*))%20or%20(not%20(last_checkin%3A*))%20or%20(unenrollment_started_at%3A*)%20)&sortField=enrolled_at&sortOrder=desc&showInactive=false&showUpgradeable=false

When I executed the curl from the command-line with that query that seemed to work.

However at this time, I am in the middle of de-registering all agents from this Elastic 8.4.2 system, so this query is irrelevant at this time.

11

As an additional data point, from my Elastic 8.4.2 system where I was having this problem, I have migrated almost 2000 elastic agents to a new system running Elastic 8.9.0.

On that system, if I run this query:

kbn:///api/fleet/agents?kuery=status:offline

this query works totally fine, and I can absolutely verify that the API response returns all offline agents.

In the Fleet UI for 8.9.0, if I toggle Status -> Offline, in the browser inspector, I see the query generated is:

/api/fleet/agents?page=1&perPage=20&kuery=status%3Aoffline&sortField=enrolled_at&sortOrder=desc&showInactive=false&showUpgradeable=false&getStatusSummary=true&withMetrics=true

So even the query generated in the UI is different in 8.9.0.

Without doing any deeper investigation, I can only conclude
that there are some UI and API differences for Fleet between 8.4.2 and 8.9.0 that make these queries behave differently.

I understand that the Fleet APIs are subject to change so that is OK.

I am in the progress of migrating off of 8.4.2 to 8.9.0, so if the Fleet APIs behave more consistently there, that is fine by me.

Thanks for the feedback and following up.

12

Great stuff, thanks for sharing the outcome. I'll remember to check the version sooner next time!

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.