Logstash http poller

Elastic Stack Logstash
1

Hi,

I am using logstash 7.13.4 and I want to use http poller for this url

http://10.10.10.10:8080/dbonequerydata/?username=user/password=password/encrypted=false/conversion=true/DT=csv -H "Content-Type: application/xml" -H "Accept: text/csv" -d @/etc/filebeat/test.xml

and my configuration looks like this

input {
  http_poller {
    urls => {
      netscout => {
        url => 'http://10.10.10.10:8080/dbonequerydata/?username=user/password=password/encrypted=false/conversion=true/DT=csv -d @/etc/filebeat/test.xml '
        headers => {
          "Content-Type" => "application/xml"
          Accept => "text/csv"
        }
      }
    }
    schedule => { cron => "40 * * * * UTC"}
  }
}

when I start the logstash it comes with error INVALID URL

I think its because the url paramater in logstash doesn't support this part ** -d @/etc/filebeat/test.xml **. But I don't know how to put it elsewhere in logstash input.

Any idea about this?

Thank you

2

If you want to send a body with the request you can use

urls => {
  netscout => {
    url => 'http://10.10.10.10:8080/dbonequerydata/?username=user/password=password/encrypted=false/conversion=true/DT=csv'
    body => ' ... '
    headers => {

You cannot use a reference to a file, you have to put the body in the logstash configuration.

Also, if you are sending a body with the request you should likely be using method => "post".

3

Hi,

I already update my configuration using body and method post and it solved error INVALID URL

input {
  http_poller {
    urls => {
      netscout => {
        method => post
        url => 'http://10.10.10.10:8080/dbonequerydata/?username=readonly/password=P@ssw0rd/encrypted=false/conversion=true/DT=csv'
        headers => {
          "Content-Type" => "application/xml"
          Accept => "text/csv"
        }
        body => '-d @/etc/filebeat/test.xml'
      }
    }
    schedule => { cron => "* * * * * UTC"}
  }
}

But it looks like the schedule is not running at all. From logstash logs no error reported.

[2022-01-27T00:47:22,994][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.6}
[2022-01-27T00:47:23,014][INFO ][logstash.inputs.http_poller][main] Registering http_poller Input {:type=>nil, :schedule=>{"cron"=>"* * * * * UTC"}, :timeout=>nil}
[2022-01-27T00:47:23,037][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-01-27T00:47:23,095][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
/home/DEV-EXTERNAL/itg/logstash-7.13.4/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated
4

Enable log.level debug and the input will log a message every time it issues a request.

Also body => '-d @/etc/filebeat/test.xml' will send that literal text with the request, not the contents of the file.