Http_poller - URL from another field

Elastic Stack Logstash
1

Hi there,
I am trying to get some data (regarding incoming calls) from an API. I am using http_poller and everything seems to work.

The Problem: The URL has a date parameter in it. I dont want to update the URL everyday, so I thought I would create a field for the current date and then just combine the URL with the date field. Unfortunately I can't get the reference to the according field working.

Here ist the .conf

input {
  http_poller {
    add_field => { "time" => "%{+YYYY-MM-dd}"}
    add_field => { "apiurl" => "https://XX.com/call_detail_records?per_page=100&date=%{time}"}
    urls => {
      call_url => {
        # Supports all options supported by ruby's Manticore HTTP client
        method => get
        url => "%{apiurl}"
        headers => {
          Authorization => "XXX"
        }
     }
    }
    # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
    schedule => { "every" => "5m" }
    codec => "json"
  }
}

And that's the resulting error:

[main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Invalid URL %{apiurl}>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.2/lib/logstash/inputs/http_poller.rb:103:in `validate_request!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.2/lib/logstash/inputs/http_poller.rb:95:in `normalize_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.2/lib/logstash/inputs/http_poller.rb:55:in `block in setup_requests!'", "org/jruby/RubyHash.java:1415:in `each'", "org/jruby/RubyEnumerable.java:886:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.2/lib/logstash/inputs/http_poller.rb:55:in `setup_requests!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.2/lib/logstash/inputs/http_poller.rb:45:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:386:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:311:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/back/logstash-calls.conf"], :thread=>"#<Thread:0x4ff385dd@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}

As always thank you for helping me out.

Kind Regards
Marvin

2

This is not going to work. The add_field is part of event decoration, and that is done after the http call has successfully completed. Also, there is no event from which you can reference fields when the HTTP request is being constructed.

If you need to do this you could use any input that supports a schedule option to create empty events, and then use mutate to do the add_field, followed by an http filter to do the API call.

3

I see. This sounds logical. I probably need to open another thread, bc this introduces a new set of weird problems. Still thank you very much for pointing this out.
Have a nice day!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.