# Logstash configuration to load next pages automatically

**URL:** https://discuss.elastic.co/t/logstash-configuration-to-load-next-pages-automatically/304457
**Category:** Logstash
**Created:** [May 11, 2022, 11:22am UTC](https://discuss.elastic.co/t/logstash-configuration-to-load-next-pages-automatically/304457 "2022-05-11T11:22:17Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)
#### Post date: [May 11, 2022, 11:22am UTC](https://discuss.elastic.co/t/logstash-configuration-to-load-next-pages-automatically/304457/1 "2022-05-11T11:22:17Z")

</div>

I am new with ELK. I apologize for mistake

I have to configure logstash in order to import data from an external source using an oai-pmh API.  
I try it using the following configuration.

```auto
input{
  http_poller{
	urls =>{
		test1=>{        
			method => get
			user => "theuserkey"
			password => "userpassword"
			url => "http://noozy.tv/oai?verb=ListRecords"
		}	
	}
	request_timeout => 60
	schedule => {"at" => "2022/05/03 10:00:00" }
codec => "json"	

  }
}

	filter {
    if !("splitted_ListRecordss" in [tags]) {
        json {
            source => "message"
        }
        split { 
            field => "ListRecords"
            add_tag => ["splitted_ListRecordss"]
        }
    }

    if ( "splitted_ListRecordss" in [tags] and [ListRecords][metas] ) {
        split {
            field => "[ListRecords][metas]"
        }
    }
}

output{
elasticsearch{
	hosts=> "localhost:9200"
	index=> "index1"
	codec => "json"
}

}

```

But I have two problems:  
The first one: **Results show that we have 3074 documents but only 20 have been imported in Elasticsearch**. An url called resumptionurl is given to be executed to get the next 20 documents until all documents are imported (then the resumptionurl is not given). \*\*I would like to configure logstash to execute the API url and get and execute the resumptionurl automatically \*\*.

Thanks for help!

---

<div class="post-metadata">

### Author: ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [May 11, 2022, 12:35pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-load-next-pages-automatically/304457/2 "2022-05-11T12:35:24Z")

</div>

I think it's not supported.  
Try with Filebeat with `httpjson` input and `response.pagination` .

> **[HTTP JSON input | Filebeat Reference \[8.2\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html)**

---

<div class="post-metadata">

### Author: ![Iss](https://avatars.discourse-cdn.com/v4/letter/i/58956e/32.png) [@Iss](https://discuss.elastic.co/u/Iss)
#### Post date: [May 11, 2022, 1:05pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-load-next-pages-automatically/304457/3 "2022-05-11T13:05:47Z")

</div>

Ok, Thank you! I will try I will try and get back .

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 8, 2022, 1:06pm UTC](https://discuss.elastic.co/t/logstash-configuration-to-load-next-pages-automatically/304457/4 "2022-06-08T13:06:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.