Elastic Defend api/endpoint/metadata: Pagination stuck when more than pageSize endpoints share the same last_checkin timestamp

Elastic Stack Elasticsearch
1

We are paginating through all endpoints using the Kibana api/endpoint/metadata API. The API does not support PIT or search_after, so we are using cursor-based pagination with
sortField=last_checkin, sortDirection=asc, and advancing via a kuery filter:

kuery: last_checkin >= ""

We always keep page=0 and advance the cursor to the last_checkin of the last endpoint in each batch, staying within Elasticsearch's 10,000-result window.

The Problem:

When more than pageSize (100) endpoints share the exact same last_checkin timestamp, the cursor never advances. Every subsequent request returns the same batch of 100 endpoints
with the same timestamp, causing the pagination loop to run indefinitely without making progress.

Example scenario:

  • Page 1 returns 100 endpoints, all with last_checkin = "2024-01-15T10:00:00Z"
  • Cursor is set to "2024-01-15T10:00:00Z" and next request uses kuery: last_checkin >= "2024-01-15T10:00:00Z"
  • Page 2 returns the same 100 endpoints again — loop is stuck

What we've tried:

  • Tracking seen agent IDs at the current cursor timestamp to skip duplicates across page boundaries — this works for overlap between two consecutive timestamps, but breaks down
    when an entire batch (or more) shares one timestamp.
  • Using > instead of >= in the kuery — this causes us to skip all endpoints at the current timestamp entirely and miss data.

Questions:

  1. Is there a way to use page + pageSize with sortField=last_checkin to paginate through a group of endpoints that all share the same timestamp (i.e., is there a stable secondary
    sort key available in this API)?

  2. Does the api/endpoint/metadata API support any tiebreaker sort field (e.g., agent.id) that could be combined with last_checkin to make each page's cursor unique?

  3. Is there a recommended approach for paginating after 10,000 endpoints using the Kibana API? We are aware of PIT + search_after on the underlying metrics-endpoint.metadata_united_default index but wanted to confirm if there's a supported API-level solution first.

  4. Is there any other API endpoint that we can use to fetch the same information?

2

Hey Rushit! Great catch on that pagination issue. Here’s the straightforward answer:

The Problem: When multiple endpoints share the same last_checkin timestamp, the cursor can’t distinguish between them, so it loops.

The Solution: The api/endpoint/metadata endpoint doesn’t support tiebreakers, but use the Elasticsearch Search API directly with PIT + search_after instead:

{
  "pit": { "id": "your_pit_id", "keep_alive": "5m" },
  "size": 100,
  "sort": [
    { "last_checkin": { "order": "asc" } },
    { "_id": { "order": "asc" } }
  ],
  "search_after": ["2024-01-15T10:00:00Z", "agent_id_here"]
}

The _id field is always unique and acts as your stable tiebreaker — this handles duplicate timestamps perfectly and works for 10K+ results.

Official docs: Paginate search results | Elasticsearch Reference

That’s the recommended approach for your use case. Let me know if you need the PIT setup details!