Logstash http_poller plugin help

Elastic Stack Logstash
1

Hello,

I am trying to use the http_poller plugin to pull data from an API end point such as the Quandl service.

First I tested the http_poller using ES url: http://localhost:9200. That worked.

Then I changed url to an quandl API. I verified that my query works using a rest client app.

I get an error message and need some help debugging.

---- my config file
input {
http_poller {
urls => {
method => get
q1 => "https://www.quandl.com/api/v3/datatables/WIKI/PRICES.json?date.gte=20170201&date.lt=20170204&ticker=MSFT&api_key=my_key_goes_here"
headers => {
Accept => "application/json"
}
}
codec => "json"
request_timeout => 60
metadata_target => "http_poller_metadata"
schedule => { cron => "* * * * * UTC"}
}
}

output {
stdout {
codec => rubydebug
}
}

Snippet of the error message:
2017-02-04T13:47:46,978][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :urls=>{"method"=>"get", "q1"=>"https://www.quandl.com/api/v3/datatables/WIKI/PRICES.json?date.gte=20170201&date.lt=20170204&ticker=MSFT&api_key=my_key_goes_here", "headers"=>{"Accept"=>"application/json"}}, :interval=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}
[2017-02-04T13:47:47,013][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: Invalid URL get>, :backtrace=>[....

Thanks,
Frank

2

The back trace is the most interesting thing.
Could you provide this ?

3

Hi Fabien,

Here is the error message with traceback:

[2017-02-04T18:29:00,410][INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :urls=>{"method"=>"get", "q1"=>"https://www.quandl.com/api/v3/datatables/WIKI/PRICES.json?date.gte=20170201&date.lt=20170204&ticker=MSFT&api_key=my_api_key”, "headers"=>{"Accept"=>"application/json"}}, :interval=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}
[2017-02-04T18:29:00,484][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: Invalid URL get>, :backtrace=>["/Users/es_usr/Downloads/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-http_poller-3.1.0/lib/logstash/inputs/http_poller.rb:163:in validate_request!'", "/Users/es_usr/Downloads/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-http_poller-3.1.0/lib/logstash/inputs/http_poller.rb:155:innormalize_request'", "/Users/es_usr/Downloads/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-http_poller-3.1.0/lib/logstash/inputs/http_poller.rb:132:in setup_requests!'", "org/jruby/RubyHash.java:1342:ineach'", "org/jruby/RubyEnumerable.java:757:in map'", "/Users/es_usr/Downloads/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-http_poller-3.1.0/lib/logstash/inputs/http_poller.rb:132:insetup_requests!'", "/Users/es_usr/Downloads/logstash-5.1.1/vendor/bundle/jruby/1.9/gems/logstash-input-http_poller-3.1.0/lib/logstash/inputs/http_poller.rb:122:in register'", "/Users/es_usr/Downloads/logstash-5.1.1/logstash-core/lib/logstash/pipeline.rb:353:instart_inputs'", "org/jruby/RubyArray.java:1613:in each'", "/Users/es_usr/Downloads/logstash-5.1.1/logstash-core/lib/logstash/pipeline.rb:352:instart_inputs'", "/Users/es_usr/Downloads/logstash-5.1.1/logstash-core/lib/logstash/pipeline.rb:228:in start_workers'", "/Users/es_usr/Downloads/logstash-5.1.1/logstash-core/lib/logstash/pipeline.rb:183:inrun'", "/Users/es_usr/Downloads/logstash-5.1.1/logstash-core/lib/logstash/agent.rb:292:in `start_pipeline'"]}
[2017-02-04T18:29:00,541][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-02-04T18:29:03,498][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}

4

It is because your "urls" config is wrong.
Up to the plugin doc, it should be :

urls => {
    q1 => {
        method => get
        url => "https://www.quandl.com/api/v3/datatables/WIKI/PRICES.json?date.gte=20170201&date.lt=20170204&ticker=MSFT&api_key=my_key_goes_here"
        headers => {
            Accept => "application/json"
        }
    }
}
5

Thanks Fabien!

6

You're welcome :slight_smile:

If your problem is solved, could you mark my previous comment as "solution" to close the topic ?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.