# Logstash http\_pollar Rest API push more than 1000 records
**URL:** https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374
**Category:** Logstash
**Created:** [November 3, 2023, 2:46pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374 "2023-11-03T14:46:45Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@puneetsharma2](https://discuss.elastic.co/u/puneetsharma2)
#### Post date: [November 3, 2023, 2:46pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/1 "2023-11-03T14:46:45Z")
Logstash http\_pollar Rest API push more than 1000 records
As we are using HTTP\_POLLAR to execute the rest API and push the response in elastic index in one go. But default only 1000 records are pushing in elastic.
How we can increase the limit or push all records in index?
How we can set the offset & limit in URL and execute the same API in multiple times to push the all response item ?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 3, 2023, 3:11pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/2 "2023-11-03T15:11:40Z")
You mean the `http_poller` input?
> [@puneetsharma2](#):
>
> How we can increase the limit or push all records in index?
I don't think that there is any limits related to the number of records, there is nothing about this in the documentation, this seems to be a limit on your API endpoint, not on the poller input.
> [@puneetsharma2](#):
>
> How we can set the offset & limit in URL and execute the same API in multiple times to push the all response item ?
With the `http_poller` input you can't, it does not support pagination.
---
### Author:  [@puneetsharma2](https://discuss.elastic.co/u/puneetsharma2)
#### Post date: [November 3, 2023, 3:29pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/3 "2023-11-03T15:29:23Z")
Thanks for reply.
I have checked Rest API in postman tool it returns 1000 + items in response, but by `http_poller` input only push 1000 records. also not set limit in Rest API URL & parameter.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 3, 2023, 3:34pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/4 "2023-11-03T15:34:39Z")
It is exactly the same request on Postman?
You need to share your Logstash pipeline and also the request you made on Postman.
I don't see anything in the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html#plugins-inputs-http_poller) that would indicate any kind of limit on the number of returned records.
---
### Author:  [@puneetsharma2](https://discuss.elastic.co/u/puneetsharma2)
#### Post date: [November 3, 2023, 3:40pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/5 "2023-11-03T15:40:55Z")
It is exactly the same request on Postman?
RE: Yes, same Rest API body & parameter are using in Postman & Logstash Pipeline.
We are passing some token & confidential data in rest API. That by I will not send logstash pipeline details.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 3, 2023, 3:41pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/6 "2023-11-03T15:41:46Z")
> [@puneetsharma2](#):
>
> We are passing some token & confidential data in rest API. That by I will not send logstash pipeline details.
You can redact it, without looking at the pipeline is not possible to troubleshoot this.
---
### Author:  [@puneetsharma2](https://discuss.elastic.co/u/puneetsharma2)
#### Post date: [November 3, 2023, 4:05pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/7 "2023-11-03T16:05:27Z")
> [@leandrojmp](#):
>
> You can redact it, without looking at the pipeline is not possible to troubleshoot this
Can we discuss in one-to-one chat or share the details in mail.
Actually, REST API is not working in your end, because its working in specific network only.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 3, 2023, 4:14pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/8 "2023-11-03T16:14:29Z")
> [@puneetsharma2](#):
>
> Can we discuss in one-to-one chat or share the details in mail.
I can't at the moment, sorry.
If you want you can remove any sensitive information and share your Logstash pipeline.
As mentioned, there is nothing in the documentation that would make limit the amount of events returned by the http\_poller input.
Also, you asked how can you set the offset, this would mean that your API paginate the response.
---
### Author:  [@puneetsharma2](https://discuss.elastic.co/u/puneetsharma2)
#### Post date: [November 3, 2023, 4:17pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/9 "2023-11-03T16:17:17Z")
> [@leandrojmp](#):
>
> If you want you can remove any sensitive information and share your Logstash pipeline.
>
> As mentioned, there is nothing in the documentation that would make limit the amount of events returned by the http\_poller input.
Logstash Pipeline :
input{
http\_poller {
urls =\> {
qradar\_rules\_url =\> {
method =\> get
urls =\> {
qradar\_rules\_url =\> {
method =\> get
url =\> "BASE\_URL"
headers =\> {
"Accept" =\> "application/json"
"Authorization" =\> "Bearer KEY"
"source-id" =\> "CC"
}
}
}
proxy =\> "PROXY\_URL"
request\_timeout =\> 60000
codec =\> "json"
schedule =\> { cron =\> "\*/5 \* \* \* \* UTC"}
}
}
filter {
split { field =\> "data" }
ruby {
code =\> '
["data"].each { |field|
event.get(field).each { |k, v|
event.set(k, v)
}
event.remove(field)
}
'
}
}
output {
elasticsearch {
hosts =\> HOST\_NAME
document\_id =\> "%{id}"
index =\> "INDEX\_NAME"
}
}
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 3, 2023, 4:25pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/10 "2023-11-03T16:25:46Z")
So, you have a custom document id with `document_id => "%{id}"`.
When you said that on Postman you had more than 1000+ items, did you validate that the you had more than 1000 unique ids on a single request?
---
### Author:  [@puneetsharma2](https://discuss.elastic.co/u/puneetsharma2)
#### Post date: [November 3, 2023, 4:38pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/11 "2023-11-03T16:38:57Z")
> [@leandrojmp](#):
>
> When you said that on Postman you had more than 1000+ items, did you validate that the you had more than 1000 unique ids on a single request?
After Remove the `document_id => "%{id}"` from pipeline and restart the Logstash. Getting same result in elastic index. (1000 items)
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 3, 2023, 5:40pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/12 "2023-11-03T17:40:52Z")
You need to provide some evidence, it is pretty hard to troubleshoot without evidences.
Since you don't want to share the endpoints you are using, even after redacting sensitive information, I suggest that you really check your API documentation to see if it does not paginate.
It seems that you are querying some QRadar API and if I'm not wrong QRadar API will paginate the response.
The `http_poller` does not limit the amount off records in the response, so this is not an issue with the input plugin.
You need also to check if this filter is correct:
```auto
filter {
split { field => "data" }
ruby {
code => '
["data"].each { |field|
event.get(field).each { |k, v|
event.set(k, v)
}
event.remove(field)
}
'
}
```
I can't validate it because you didn't share any sample data, so it is not possible to know what will be the output of this.
Also, try to replicate the same request using a tool like `curl`.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 3, 2023, 6:35pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/13 "2023-11-03T18:35:20Z")
> [@puneetsharma2](#):
>
> code =\> '
> ["data"].each { |field|
> event.get(field).each { |k, v|
> event.set(k, v)
> }
> event.remove(field)
> }
> '
It took me a while to understand that. It would be simpler to write it as
```
code => '
event.remove("data").each { |k, v|
event.set(k, v)
}
'
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 1, 2023, 6:35pm UTC](https://discuss.elastic.co/t/logstash-http-pollar-rest-api-push-more-than-1000-records/346374/14 "2023-12-01T18:35:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.