# Elastic Agent, Custom API Integration - GET Next URL from JSON response

**URL:** <https://discuss.elastic.co/t/elastic-agent-custom-api-integration-get-next-url-from-json-response/335104>\
**Category:** Elastic Agent\
**Created:** [June 2, 2023, 3:57pm UTC](https://discuss.elastic.co/t/elastic-agent-custom-api-integration-get-next-url-from-json-response/335104 "2023-06-02T15:57:05Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mark\_Campbell](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_campbell/32/121792_2.png) [@Mark\_Campbell](https://discuss.elastic.co/u/Mark_Campbell)\
**Post date:** [June 2, 2023, 3:57pm UTC](https://discuss.elastic.co/t/elastic-agent-custom-api-integration-get-next-url-from-json-response/335104/1 "2023-06-02T15:57:05Z")

</div>

I'm using ES, Kibana and Agent version 8.8.0.

I can use the Custom API Integration to get the JSON response from the API.

API URL:

`https://example.com/api/data/?page=1`

JSON Response:

```auto
{
  "data": [
    {
      "attributes": {
        "autodetected": false,
                             }
---SNIP---
    }
  ],
  "links": {
    "first": "https://example.com/api/data/?page=1",
    "last": "https://example.com/api/data/?page=4",
    "next": "https://example.com/api/data/?page=2",
    "self": "https://example.com/api/data/?page=1"
  },
  "meta": {
    "api_version": "10",
    "pagination": {
      "page": 1,
      "perPage": 50,
      "totalPages": 4
    },
    "version": "6.0.0"
  }
}

```

So my question is, what is the best way to "chain" or "step" through all the pages to get all the API data?

I would like the Custom API integration to read the "next" page and get the data:

`https://example.com/api/data/?page=2`  
`https://example.com/api/data/?page=3`  
`https://example.com/api/data/?page=4`  
etc

I've tried a number of the examples listed online, but I'm not getting it right:

```auto
    - step:
      request.url: $.nextURL
      request.method: GET
      replace_with: '$.nextURL,.last_response.message.links.next'

or

    - step:
      request.url: $.links[:].next
      request.method: GET
      replace: $.links[:].next

or

- step:
    request.url: https://example.com/api/data/?page=$.records[:].id
    request.method: GET
    replace: $.records[:].id

```

I keep on hitting errors, like:

```auto
[elastic_agent][error] Unit state changed httpjson-default-httpjson-generic-7b353d00-0059-11ee-8a74-1f22250d20c6 (CONFIGURING-><mark>FAILED</mark>): [<mark>failed</mark> to reloading inputs: 1 error: Error creating runner from config: type mismatch accessing 'chain.0.step.request.url']

[elastic_agent][error] Unit state changed httpjson-default-httpjson-generic-7b353d00-0059-11ee-8a74-1f22250d20c6 (CONFIGURING-><mark>FAILED</mark>): [<mark>failed</mark> to reloading inputs: 1 error: Error creating runner from config: both step & while blocks in a chain cannot be empty accessing config]

[elastic_agent][error] Unit state changed httpjson-default-httpjson-generic-7b353d00-0059-11ee-8a74-1f22250d20c6 (CONFIGURING-><mark>FAILED</mark>): [<mark>failed</mark> to reloading inputs: 1 error: Error creating runner from config: parse "$.links[:].next": first path segment in URL cannot contain colon accessing 'chain.0.step.request.url']

```

---

<div class="post-metadata">

**Author:** ![Mark\_Campbell](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_campbell/32/121792_2.png) [@Mark\_Campbell](https://discuss.elastic.co/u/Mark_Campbell)\
**Post date:** [June 7, 2023, 1:24pm UTC](https://discuss.elastic.co/t/elastic-agent-custom-api-integration-get-next-url-from-json-response/335104/2 "2023-06-07T13:24:35Z")

</div>

To Answer My Own Question, this is what worked:

```auto
response pagination
- set:
    target: url.value
    value: >- 
        [[- if ne .last_response.body.links.next .last_response.body.links.first -]]
            [[- .last_response.body.links.next -]]
        [[- end -]]
    fail_on_template_error: true

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2023, 1:24pm UTC](https://discuss.elastic.co/t/elastic-agent-custom-api-integration-get-next-url-from-json-response/335104/3 "2023-07-05T13:24:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.