Pagination using Filebeat httpjson how to reference the 'next' link in results

[Jez1 (Jeremy)]

Hello, I'm pretty new to using filebeat and currently working out how to use the httpjson input to pull data from a REST API.

I've worked out how to pull the response based on the time period I need and in the correct time format.
What I'm having trouble with is the pagination of the response.
In the response there is a bunch of links that show what URL to use to get the next page of results. I'm not sure how to reference it when using the 'response.pagination' in the filebeat config.

Any help or pointers in the correct direction are hugely appreciated.

For reference:
My current filebeat config, the 'response.pagination' I was playing with based on suggestions I found online, but don't really understand it.

filebeat.inputs:
- type: httpjson
  config_version: "2"
  interval: 1m
  enabled: true

  request.url: "https://api.safenetid.com/api/v1/tenants/xxxxx/logs"
  request.method: GET

  request.transforms:
    - set:
        target: header.apikey
        value: "xxxxx"
    - set:
        target: url.params.since
        value: '[[ formatDate (now (parseDuration "-6h")) "2006-01-02T15:04:05.000Z" ]]'
    - set:
        target: header.Content-Type
        value: application/json


  response.pagination:
    - set:
        target: "cursor.next_url"
        value: "{{.links.next}}"

output.console:
  pretty: true

Cutdown example of the results.

 {
  "page": {
    "totalItems": 10000,
    "pageMarker": 1707655139940,
    "pageSize": 1000,
    "totalPages": 10,
  },
<actual log entries go here>
  "links": {
    "first": "https://api.safenetid.com/api/v1/tenants/12345/logs?since=2024-02-11T10:50:00.000Z",
    "self": "https://api.safenetid.com/api/v1/tenants/12345/logs?since=2024-02-11T10:50:00.000Z",
    "next": "https://api.safenetid.com/api/v1/tenants/12345/logs?since=2024-02-11T10:50:00.000Z&marker=1707655139940"
  },
  "itemType": "LogEvent"
}

And the documentation for the API I'm using is here, in case someone has a better suggestion for how to achieve this.
https://api.safenetid.com/swagger/index.html