Pagination Logic in input of HTTP Poller plugin

I have to bulk import data via LogStash as below:
Source: Restful Get APIs
Destination : ElasticSearch

My .conf file looks as below:
input {
http_poller {
urls => {
test1 => {
method => get
url => "https://someurl/movies?limit=100&offset=0"
headers => {
Accept => "application/json"

This fetches 100 records at a time.
But, i have more than 5000 records and need to apply pagination logic in input of http-poller.

Please help how can i apply pagination logic here.


The http_poller has no support for paginated requests.

Can you please provide any input on how to handle this scenario of pagination ?

I'd patch the http_poller plugin or write a custom plugin. An external custom script (run via an exec input) would be another option.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.