# Issue using http\_poller to connect to streaming http api

**URL:** <https://discuss.elastic.co/t/issue-using-http-poller-to-connect-to-streaming-http-api/270434>\
**Category:** Logstash\
**Created:** [April 17, 2021, 12:18am UTC](https://discuss.elastic.co/t/issue-using-http-poller-to-connect-to-streaming-http-api/270434 "2021-04-17T00:18:26Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![metalshanked](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/metalshanked/32/87159_2.png) [@metalshanked](https://discuss.elastic.co/u/metalshanked)\
**Post date:** [April 17, 2021, 12:18am UTC](https://discuss.elastic.co/t/issue-using-http-poller-to-connect-to-streaming-http-api/270434/1 "2021-04-17T00:18:26Z")

</div>

Hi Is there a way to implement a setting similar to Python requests "stream=True" setting in the Logstash http\_poller plugin. I am trying to pull streaming data from Shodan using logstash and get "no upstream" errors.  
I understand this happens when the "stream" type setting is not enabled which tells it to wait until the content is obtained (i believe thats how python requests stream=True works)

Here is my Logstash config

```auto
input {
      http_poller {
        urls => {
          test1 => {
            method => get
            url => "https://stream.shodan.io/shodan/banners?key=API_KEY&t=json"
            headers => {
              "Accept-Encoding" => "gzip, deflate"
             "Accept" => "*/*"
            "Connection" => "keep-alive"
            }
         }
        }
        schedule => {"every" => "1s"}
        codec => json
        metadata_target => "http_poller_metadata"
      }
    }
    output {
    	stdout => {codec =>rubydebug}
    }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 17, 2021, 12:49am UTC](https://discuss.elastic.co/t/issue-using-http-poller-to-connect-to-streaming-http-api/270434/2 "2021-04-17T00:49:17Z")

</div>

There is an [open issue](https://github.com/logstash-plugins/logstash-input-http_poller/issues/101) asking for the input to support streaming APIs, so I doubt it can currently do so.

---

<div class="post-metadata">

**Author:** ![metalshanked](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/metalshanked/32/87159_2.png) [@metalshanked](https://discuss.elastic.co/u/metalshanked)\
**Post date:** [April 17, 2021, 1:14am UTC](https://discuss.elastic.co/t/issue-using-http-poller-to-connect-to-streaming-http-api/270434/3 "2021-04-17T01:14:36Z")

</div>

@Badger Thanks for the info and link! Seems like it is a long standing request.  
I tried adding a "Transfer-Encoding" =\> "chunked" header but weirdly it throws a "Transfer-Encoding" header already present error but i don't see where it is getting added apart from the one i declared

---

<div class="post-metadata">

**Author:** ![bagginstyrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bagginstyrone/32/86981_2.png) [@bagginstyrone](https://discuss.elastic.co/u/bagginstyrone)\
**Post date:** [April 17, 2021, 8:13am UTC](https://discuss.elastic.co/t/issue-using-http-poller-to-connect-to-streaming-http-api/270434/4 "2021-04-17T08:13:05Z")

</div>

> [@Badger](#):
>
> There is an [open issue](https://github.com/logstash-plugins/logstash-input-http_poller/issues/101) asking for the input to support streaming APIs, so I doubt it can currently do so.

@Badger Thank you so much for your response. It also helped me to resolve the API issue. 🙂

---

<div class="post-metadata">

**Author:** ![metalshanked](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/metalshanked/32/87159_2.png) [@metalshanked](https://discuss.elastic.co/u/metalshanked)\
**Post date:** [April 17, 2021, 4:12pm UTC](https://discuss.elastic.co/t/issue-using-http-poller-to-connect-to-streaming-http-api/270434/5 "2021-04-17T16:12:22Z")

</div>

@bagginstyrone How did you resolve this issue?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 15, 2021, 4:12pm UTC](https://discuss.elastic.co/t/issue-using-http-poller-to-connect-to-streaming-http-api/270434/6 "2021-05-15T16:12:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.