Transfer json format data from external service to elasticseach

trying to transfer data in json format which in external service to elasticseach.
data provide page by page. next page define in previous page
how to transfer data from all pages to elasticseach

sample json file

count: 565946
next: "http://test.data.tech/api/static/v1/hotels/?format=json&page=2"
previous: "null"
results
0: {…}
1:
address: "Keizersgracht 164"
availability_score: 0.000837698434291508
book_score: 501
checkin_from: "14:00"

used http_poller input plugin but it transfer only data in first page.

input {
http_poller {
urls => {

  test2 => {
    method => GET 
    user => "xxxx"
    password => "xxxx"
    url => "http://test.data.tech/api/static/v1/hotels"
   
 }
}
request_timeout => 60
schedule => { cron => "* * * * * UTC"}
codec => "json"

}

}

Is there way to transfer data in all pages to a index

You could write a script file for a ruby filter which walks down a chain of linked pages and returned an array of events, but to be honest this use case is not a good fit for logstash.

Thanks for the guide. by the way are there any other methods to do this.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.