# Logstash input plugin http\_poller only loading 300 records in a run

**URL:** <https://discuss.elastic.co/t/logstash-input-plugin-http-poller-only-loading-300-records-in-a-run/332281>\
**Category:** Logstash\
**Created:** [May 2, 2023, 1:10pm UTC](https://discuss.elastic.co/t/logstash-input-plugin-http-poller-only-loading-300-records-in-a-run/332281 "2023-05-02T13:10:47Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhisarika\_verma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhisarika_verma/32/112894_2.png) [@abhisarika\_verma](https://discuss.elastic.co/u/abhisarika_verma)\
**Post date:** [May 2, 2023, 1:10pm UTC](https://discuss.elastic.co/t/logstash-input-plugin-http-poller-only-loading-300-records-in-a-run/332281/1 "2023-05-02T13:10:47Z")

</div>

I am trying to use the http\_poller input plugin to load data from sap CDC into Elasticsearch.  
But Logstash input plugin http\_poller is only loading 300 records in a run.

1. How can I load bulk data using the http\_poller plugin?
2. If the bulk load is impossible, how can I dynamically pass an incremental date value for "created" in the below query for every run in the schedule?

````auto
input {
http_poller {
  urls => {
    test_url =>{
      url => "${cdcurl}"
      method => post
      user => "${devuser}"
      password => "${password}"
      headers => {
                "Accept" => "application/json"
              }
      params => {
          query => "select * from accounts where created >= '2022-09-17T15:00:00.010Z'"
        }
      }
  }
  codec => "json"
  request_timeout => 60
  schedule => { "every" => "20s" }
}
}

filter {
 json {
    source => "message"
  }
  split {
    field => "[results]"
  }
  prune {
        whitelist_names => ["^results$"]
      }
}

output {
    elasticsearch {
        cloud_id => '${ELASTICSEARCH_CLOUD_ID}'
        cloud_auth => '${ELASTICSEARCH_CLOUD_AUTH}'
        index => "sapcdc"
        action => "index"
        document_id => "%{[results][UID]}"
    }
    stdout { codec => rubydebug }
}```
````

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 30, 2023, 1:11pm UTC](https://discuss.elastic.co/t/logstash-input-plugin-http-poller-only-loading-300-records-in-a-run/332281/2 "2023-05-30T13:11:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.