# Pagination on logstash http filter
**URL:** https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320
**Category:** Logstash
**Created:** [February 1, 2024, 3:38pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320 "2024-02-01T15:38:36Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Johnson\_will](https://discuss.elastic.co/u/Johnson_will)
#### Post date: [February 1, 2024, 3:38pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320/1 "2024-02-01T15:38:36Z")
I am generating a token by using http\_poller input plugin, using the generated token in http filter part and splitting the fields from results, Can anyone please suggest me on Pagination.
As the api results are around 4000,
> Blockquote
`Preformatted text`{
"total": 4425,
"results": [
{
"\_id": {
"$oid": "65aac96f90ebf09ba386170e"
},................................................ },................................................
`````auto
`> Blockquote`
type or paste code here
````Preformatted text`
my logstash pipeline:
input {
http_poller {
urls => {
generatetoken => {
method => post
body => '{ "userID" :"${USER}", "userPWD" : "${PWD}" }'
headers => {
"Content-Type" => "application/json"
}
url => "https:// ********** /auth/login"
request_timeout => 60
}
}
truststore => "/usr/share/logstash/config/Kafka/cacerts"
truststore_password => "changeit"
schedule => { cron => "* * * * * UTC"}
metadata_target => "my-token"
}
}
filter{
http {
body_format => "json"
follow_redirects => false
verb => "POST"
headers => { "Content-Type" => "application/json"
"Authorization" => "%{token}" }
body => { "appCode" => " *****"}
url => "https:// *****"
truststore => "/usr/share/logstash/config/Kafka/cacerts"
truststore_password => "changeit"
target_body => "[api_response]"
# target_headers => "[@metadata][token]"
}
split { field => "[api_response][results]" }
}
elasticsearch {
hosts => ["${ES_HOST}"]
user => "${USER}"
password => "${PWD}"
ilm_enabled => true
ilm_policy => " ***** _lifecycle_policy"
ilm_rollover_alias => "${INDEX_ENV}-jobhistory-api"
document_id => "%{[api_response][results][_id][$oid]}"
ssl => true
ssl_certificate_verification => true
cacert => "/usr/share/logstash/config/SSL/${SSL_ENV}/elastic-cert.cer"
}
}`Preformatted text`
`````
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 1, 2024, 4:58pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320/2 "2024-02-01T16:58:01Z")
> [@Johnson\_will](#):
>
> Can anyone please suggest me on Pagination.
The `http_poller` input plugin does not support pagination, you can't do that with Logstash.
An alternative would be using Filebeat in front of Logstash making the requests since the `httpjson` input of filebeat supports [pagination](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-httpjson.html#filebeat-input-httpjson).
---
### Author:  [@Johnson\_will](https://discuss.elastic.co/u/Johnson_will)
#### Post date: [February 1, 2024, 6:55pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320/3 "2024-02-01T18:55:15Z")
Thanks [Leandro Pereira] for your reply.
I wanted to achieve this by logstash, i sthere any other way around using logstash?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 1, 2024, 8:35pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320/4 "2024-02-01T20:35:11Z")
> [@Johnson\_will](#):
>
> I wanted to achieve this by logstash, i sthere any other way around using logstash?
As mentioned no, the `http_poller` filter does not support pagination.
---
### Author:  [@Johnson\_will](https://discuss.elastic.co/u/Johnson_will)
#### Post date: [February 2, 2024, 5:07pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320/5 "2024-02-02T17:07:33Z")
I was referring to the below link
> [@Paginantion in logstash](https://discuss.elastic.co/t/paginantion-in-logstash/323385):
>
> Hello there ! I am fetching data from vRops api using http\_poller. Is there any way to achieve pagination in logstash ?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 2, 2024, 6:04pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320/6 "2024-02-02T18:04:41Z")
Hello,
Not sure what you are referring to, the link you shared is unrelated to the `http_poller` input.
As mentioned the `http_poller` input does not support pagination, there is nothing native in logstash that supports pagination.
You may be able to kind of replicate a pagination combining some logic and filters, as the link you mentioned, but I don't think this is the best approach.
Logstash is not the best tool for this, it would be easier to use Filebeat in this case to make the polling or write a quick python script that would make the request and save into a file for Logstash to process.
---
### Author:  [@Johnson\_will](https://discuss.elastic.co/u/Johnson_will)
#### Post date: [February 2, 2024, 6:12pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320/7 "2024-02-02T18:12:09Z")
Thanks a lot for your quick response,
Filebeat doesn't work for me as I am fetching the results by an api.
Do you have an example for phython script which saves a file for logstash to process.
Appreciate your help.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 2, 2024, 6:40pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320/8 "2024-02-02T18:40:15Z")
> [@Johnson\_will](#):
>
> Filebeat doesn't work for me as I am fetching the results by an api.
Did you check the documentation for the `httpjson` input that was shared before? This input does exactly that, it can make requests to APIs.
> [@Johnson\_will](#):
>
> Do you have an example for phython script which saves a file for logstash to process.
I do not, but you can easily find some searching on Google, this is unrelated to Logstash, you just need to make a request to your API and then save it on a file.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 1, 2024, 6:40pm UTC](https://discuss.elastic.co/t/pagination-on-logstash-http-filter/352320/9 "2024-03-01T18:40:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.