# \_http\_request\_failure in logstash while using http\_poller **URL:** \ **Category:** Logstash\ **Created:** [October 5, 2015, 1:48pm UTC](https://discuss.elastic.co/t/-http-request-failure-in-logstash-while-using-http-poller/31637 "2015-10-05T13:48:37Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Navneet\_Mathpal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/navneet_mathpal/32/3677_2.png) [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Post date:** [October 5, 2015, 1:48pm UTC](https://discuss.elastic.co/t/-http-request-failure-in-logstash-while-using-http-poller/31637/1 "2015-10-05T13:48:37Z")
Hi , I am getting some json documents from http\_poller input { http\_poller { urls =\> { "localhost" =\> "[http://localhost:9200](http://localhost:9200)" } interval =\> 10 After I reads all the document It shows \_http\_request\_failure 1 . **Is it because of logstash pinging the URL in every 10 sec and if it does not get any new doc over there it will show this \_http\_request\_failure ?** 2. **If any new doc get updated in URL will http\_poller will be able to take it in real time ? without taking the older docs ?** Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 5, 2015, 1:53pm UTC](https://discuss.elastic.co/t/-http-request-failure-in-logstash-while-using-http-poller/31637/2 "2015-10-05T13:53:31Z")
Not sure what you mean by older and newer docs. [http://localhost:9200](http://localhost:9200) will only return Elasticsearch's rarely-changing status document and the http\_poller just makes an HTTP request and passes the results to Logstash. ---
**Author:** ![Navneet\_Mathpal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/navneet_mathpal/32/3677_2.png) [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Post date:** [October 5, 2015, 2:01pm UTC](https://discuss.elastic.co/t/-http-request-failure-in-logstash-while-using-http-poller/31637/3 "2015-10-05T14:01:56Z")
I mean , if I have 200 json docs availble in my url "www.examplecom/jsonfile". After running the http\_poller I will get 200 json docs , what if one new doc get updated , will http\_poller only take that new updated doc or it will take all the 201 docs again ? why this \_http\_request\_failure comes ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 5, 2015, 2:13pm UTC](https://discuss.elastic.co/t/-http-request-failure-in-logstash-while-using-http-poller/31637/4 "2015-10-05T14:13:50Z")
Okay, so [http://localhost:9200](http://localhost:9200) was just a randomly picked URL? That was not obvious. > After running the http\_poller I will get 200 json docs , what if one new doc get updated , will http\_poller only take that new updated doc or it will take all the 201 docs again ? http\_poller does not maintain any state, i.e. it has no idea of what documents are new. > why this httprequest\_failure comes ? The resulting event's `@metadata` field (normally not emitted by outputs but available with e.g. `stdout { codec => rubydebug }`) should contain details about the failure. ---
**Author:** ![Navneet\_Mathpal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/navneet_mathpal/32/3677_2.png) [@Navneet\_Mathpal](https://discuss.elastic.co/u/Navneet_Mathpal)\ **Post date:** [October 5, 2015, 5:30pm UTC](https://discuss.elastic.co/t/-http-request-failure-in-logstash-while-using-http-poller/31637/5 "2015-10-05T17:30:13Z")
Thank you @magnusbaeck So we can handle the repetitive doc using document\_id. 1 .If I have 1m json records available in my URL , AND the url is referencing in every 10 sec .(and let us suppose http\_poller reading 100 doc/sec ) its means that htttp\_poller can never read the whole file ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 5, 2015, 5:55pm UTC](https://discuss.elastic.co/t/-http-request-failure-in-logstash-while-using-http-poller/31637/6 "2015-10-05T17:55:34Z")
Your architecture doesn't sound very sustainable, at least not with the update frequency you have in mind. Those 1 million documents have to weigh, what, 100 MB or more? You _really_ want to do some kind of incremental polling ("give me everything that's been updated since X") or that the origin system sends updates to a broker that you can read from. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:27am UTC](https://discuss.elastic.co/t/-http-request-failure-in-logstash-while-using-http-poller/31637/7 "2017-07-06T05:27:19Z")