Pagination Logic in input of HTTP Poller plugin

Elastic Stack Logstash
1

I have to bulk import data via LogStash as below:
Source: Restful Get APIs
Destination : ElasticSearch

My .conf file looks as below:
input {
http_poller {
urls => {
test1 => {
method => get
url => "https://someurl/movies?limit=100&offset=0"
headers => {
Accept => "application/json"
}
}
}

This fetches 100 records at a time.
But, i have more than 5000 records and need to apply pagination logic in input of http-poller.

Please help how can i apply pagination logic here.

Thanks.

2

The http_poller has no support for paginated requests.

3

Can you please provide any input on how to handle this scenario of pagination ?

4

I'd patch the http_poller plugin or write a custom plugin. An external custom script (run via an exec input) would be another option.