Httop Poller Input Plugin - Need help

[Anusha_Kusanghi]

Hie Everyone,
HTTP poller plugin when used fetches only 1000 records from the URL triggered even though the actual response has more than 1000 events. ​

here is the config file :
input { http_poller { urls => { audit_logs => { method => get user => "user1" password => "pass1" url => "https://t123.test.com/gateway/mar.stra/v1/logs/8940a8d7-162?startDate=1609455600&endDate=1640754001" headers => { Accept => "*/*" "Content-Type" => "application/json" } } } truststore => "/etc/pki/java/cacerts" truststore_password => "test123" request_timeout => 600 socket_timeout => 600 schedule => { cron => "58 * * * * UTC"} codec => "json" metadata_target => "http_poller_metadata" } }
I used the start date and end date in epoch timestamp with values 01/01/2021 to 29/12/2021. There is data existing for the entire year but I only receive the first 1000 records from the response. Can you please let us know what can be the reason and what can be done to get all the events loaded to logstash from HTTP poller response?

TIA

[leandrojmp (Leandro Pereira)]

This seems to be a limitation from the endpoint you are calling, not logstash as there is nothing in the filter that would limit the number of events received.

Can you get more than 1000 events when you make the same request without using logstash? Try to use Postman or a Python script to check that.

[Anusha_Kusanghi]

thanks @leandrojmp that actually helped.