Hello,
I am having 1 Billion data coming from an http API. The Api has pagination so that I can fetch data in batches. How can I implement that pagination in logstash using http_poller plugin...
My config file structure looks like this:
input {
http_poller {
urls => {
api_url=> {
method => "POST"
url => "api url"
headers => {
"Content-Type" => "application/json"
}
body=> '{"page":1 , "pagesize":20}'
}
}
request_timeout => 120
schedule => {every => "1m"}
codec => "json"
}
}
filter {
mutate {
}
}
output {
elasticsearch {
hosts => ["https://localhost:9200"]
user => "---"
password => "----"
index => "my-index"
document_id => "%{--}"
doc_as_upsert => true
action => "update"
}
stdout { codec => rubydebug }
}
I want to call this api multiple times with different page and pagesize parameters untill all the pages are reached.