Httpjson pagination querystring value + 1

Elastic Stack Beats
1

Trying to use httpjson integration to scrape an api of data, using filebeat.8.2.2

The api takes query parameters PageSize=1000&PageNumber=1, where I'm supposed to keep adding PageNumber +1 until I get a 404 error.

However I can't figure out how to configure.

- set:
    target: url.params.PageNumber
    value: [[add .last_response.url.params.PageNumber 1]]
    fail_on_template_error: true

gives error: [elastic_agent.filebeat][error] Error creating runner from config: fail to unpack the set configuration: type mismatch accessing 'response.pagination.0.set.value' accessing 'response'

if I change value with a '

- set:
    target: url.params.PageNumber
    value: '[[add .last_response.url.params.PageNumber 1]]'
    fail_on_template_error: true

gives error: [elastic_agent.filebeat][error] Error creating runner from config: fail to unpack the set configuration: type mismatch accessing 'response.pagination.0.set.value' accessing 'response'

How can I set the pagination rules to keep increasing until done?
image

2

Hi @frolko,

There are a couple of mistakes on your syntax:

  1. To access the URL parameter you have to use the Get function: [[.last_response.ur.params.Get "foo"]]
  2. URL parameters are read as string, so you have to convert them to int before doing mathematical operations with it.

A fixed version of your configuration is:

      - set:
          target: url.params.PageNumber
          value: '[[ add (toInt (.last_response.url.params.Get "PageNumber")) 1 ]]'
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.