Elastic Agent, Custom API Integration - GET Next URL from JSON response

Elastic Stack Elastic Agent
1

I'm using ES, Kibana and Agent version 8.8.0.

I can use the Custom API Integration to get the JSON response from the API.

API URL:

https://example.com/api/data/?page=1

JSON Response:

{
  "data": [
    {
      "attributes": {
        "autodetected": false,
                             }
---SNIP---
    }
  ],
  "links": {
    "first": "https://example.com/api/data/?page=1",
    "last": "https://example.com/api/data/?page=4",
    "next": "https://example.com/api/data/?page=2",
    "self": "https://example.com/api/data/?page=1"
  },
  "meta": {
    "api_version": "10",
    "pagination": {
      "page": 1,
      "perPage": 50,
      "totalPages": 4
    },
    "version": "6.0.0"
  }
}

So my question is, what is the best way to "chain" or "step" through all the pages to get all the API data?

I would like the Custom API integration to read the "next" page and get the data:

https://example.com/api/data/?page=2
https://example.com/api/data/?page=3
https://example.com/api/data/?page=4
etc

I've tried a number of the examples listed online, but I'm not getting it right:

    - step:
      request.url: $.nextURL
      request.method: GET
      replace_with: '$.nextURL,.last_response.message.links.next'

or

    - step:
      request.url: $.links[:].next
      request.method: GET
      replace: $.links[:].next

or

- step:
    request.url: https://example.com/api/data/?page=$.records[:].id
    request.method: GET
    replace: $.records[:].id

I keep on hitting errors, like:

[elastic_agent][error] Unit state changed httpjson-default-httpjson-generic-7b353d00-0059-11ee-8a74-1f22250d20c6 (CONFIGURING-><mark>FAILED</mark>): [<mark>failed</mark> to reloading inputs: 1 error: Error creating runner from config: type mismatch accessing 'chain.0.step.request.url']

[elastic_agent][error] Unit state changed httpjson-default-httpjson-generic-7b353d00-0059-11ee-8a74-1f22250d20c6 (CONFIGURING-><mark>FAILED</mark>): [<mark>failed</mark> to reloading inputs: 1 error: Error creating runner from config: both step & while blocks in a chain cannot be empty accessing config]

[elastic_agent][error] Unit state changed httpjson-default-httpjson-generic-7b353d00-0059-11ee-8a74-1f22250d20c6 (CONFIGURING-><mark>FAILED</mark>): [<mark>failed</mark> to reloading inputs: 1 error: Error creating runner from config: parse "$.links[:].next": first path segment in URL cannot contain colon accessing 'chain.0.step.request.url']
2

To Answer My Own Question, this is what worked:

response pagination
- set:
    target: url.value
    value: >- 
        [[- if ne .last_response.body.links.next .last_response.body.links.first -]]
            [[- .last_response.body.links.next -]]
        [[- end -]]
    fail_on_template_error: true
1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.