Http_poller give time out error on data response of 80 K records

[rkhapre (RK)]

Hi All

I have http end point which gives data of 80 K., after running the pipeline i get the error as

"name"=>"http_request", "error"=>"Read timed out"}.

I have tried all the options of http_poller input plugin like

                    connect_timeout => 10000
			request_timeout => 90000
			automatic_retries => 10

If i try to pull with same end point less amount of records like 3000, then it works perfectly.

Please let me know what settings required to pull data in 1m or 80K records

[rkhapre (RK)]

Hi All

Any one faced this problem, i think this is very unique problem with large dataset.