Issue using http_poller to connect to streaming http api

Elastic Stack Logstash
#1

Hi Is there a way to implement a setting similar to Python requests "stream=True" setting in the Logstash http_poller plugin. I am trying to pull streaming data from Shodan using logstash and get "no upstream" errors.
I understand this happens when the "stream" type setting is not enabled which tells it to wait until the content is obtained (i believe thats how python requests stream=True works)

Here is my Logstash config

input {
      http_poller {
        urls => {
          test1 => {
            method => get
            url => "https://stream.shodan.io/shodan/banners?key=API_KEY&t=json"
            headers => {
              "Accept-Encoding" => "gzip, deflate"
             "Accept" => "*/*"
            "Connection" => "keep-alive"
            }
         }
        }
        schedule => {"every" => "1s"}
        codec => json
        metadata_target => "http_poller_metadata"
      }
    }
    output {
    	stdout => {codec =>rubydebug}
    }
#2

There is an open issue asking for the input to support streaming APIs, so I doubt it can currently do so.

1 Like
#3

@Badger Thanks for the info and link! Seems like it is a long standing request.
I tried adding a "Transfer-Encoding" => "chunked" header but weirdly it throws a "Transfer-Encoding" header already present error but i don't see where it is getting added apart from the one i declared

#4

@Badger Thank you so much for your response. It also helped me to resolve the API issue. :slightly_smiling_face:

#5

@bagginstyrone How did you resolve this issue?

#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.