How to sync data from htttp_poller

Elastic Stack Logstash
1

Hello,
I am having 1 Billion data coming from an http API. The Api has pagination so that I can fetch data in batches. How can I implement that pagination in logstash using http_poller plugin...
My config file structure looks like this:

input {
  http_poller {
    urls => {
	api_url=> {
        method => "POST"
        url => "api url"
		headers => {
                    "Content-Type" => "application/json"
                }
        body=> '{"page":1 , "pagesize":20}'
      }
	}	
	request_timeout => 120
	schedule =>  {every => "1m"}
	codec => "json"
  }
}
   filter { 
	  mutate {
	  }
	}
output {
		elasticsearch {
			hosts => ["https://localhost:9200"]
			user => "---"
			password => "----"
			index => "my-index"
			document_id => "%{--}"
			doc_as_upsert => true
			action => "update"
		}
  stdout { codec => rubydebug }
}

I want to call this api multiple times with different page and pagesize parameters untill all the pages are reached.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.