Http_poller error when building XML/SOAP request

Hi there, I'm trying to use the http_poller input to periodically generate an HTTP/XML/SOAP requests but I'm getting "Invalid URL post" back - Logstash doesn't even seem to be building the request:

Pipeline aborted due to error {:pipeline_id=>"ucm-get-perfmon", :exception=>#<LogStash::ConfigurationError: Invalid URL post>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:103:in `validate_request!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:95:in `normalize_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:55:in `block in setup_requests!'", "org/jruby/ `each'", "org/jruby/ `map'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:55:in `setup_requests!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:45:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:199:in `block in register_plugins'", "org/jruby/ `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:198:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:303:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:259:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:153:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:108:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipelines/ucm-get-perfmon.conf"], :thread=>"#<Thread:0x7653662d run>"}

My config for logstash looks like this:

input {
    http_poller {
        urls => {
            method => post
            url => ""
            auth => {
                user => "user"
                password => "password"
            headers => {
                "Content-Type" => "text/xml"
                "SOAPAction" => "CUCM:DB ver=12.5"
            body => '<!--LogCollection API - SelectLogFiles - Request--><soapenv:Envelope xmlns:soapenv="" xmlns:soap=""><soapenv:Header/><soapenv:Body><soap:selectLogFiles><soap:FileSelectionCriteria><soap:ServiceLogs><soap:item></soap:item></soap:ServiceLogs><soap:SystemLogs><soap:item>Cisco RIS Data Collector PerfMonLog</soap:item></soap:SystemLogs><soap:SearchStr></soap:SearchStr><soap:Frequency>OnDemand</soap:Frequency><soap:JobType>PushtoSFTPServer</soap:JobType><soap:ToDate></soap:ToDate><soap:FromDate></soap:FromDate><soap:TimeZone></soap:TimeZone><soap:RelText>Days</soap:RelText><soap:RelTime>5</soap:RelTime><soap:Port>22</soap:Port><soap:IPAddress></soap:IPAddress><soap:UserName>username</soap:UserName><soap:Password>password</soap:Password><soap:ZipInfo>false</soap:ZipInfo><soap:RemoteFolder>/data/tracelogger/test/</soap:RemoteFolder></soap:FileSelectionCriteria></soap:selectLogFiles></soapenv:Body></soapenv:Envelope>'
        request_timeout => 60
        # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
        schedule => { cron => "* * * * * UTC" }

output {
  stdout {
    codec => rubydebug

I can get the request working OK using postman and it looks like this:

POST /logcollectionservice2/services/LogCollectionPortTypeService?wsdl HTTP/1.1
SOAPAction: CUCM:DB ver=12.5 
Content-Type: text/xml
Authorization: Basic dHJhY2Vsb2dnZXIuYXhsLXJvLmFkbWluOnp5bU9pRGVybk1vcg==
User-Agent: PostmanRuntime/7.20.1
Accept: */*
Cache-Control: no-cache
Postman-Token: 65b97d05-1e4f-44cd-af48-935989a062ef,47702e32-3cf2-4668-9a79-8c82e26a5e9f
Accept-Encoding: gzip, deflate
Content-Length: 1422
Cookie:; JSESSIONID=A474438A26B671FEBE53C0DF917B0A9E; JSESSIONIDSSO=DC7D5E497DC11A2B3EF86107207F278B
Connection: keep-alive
cache-control: no-cache

<!--LogCollection API - SelectLogFiles - Request-->
<soapenv:Envelope xmlns:soapenv="" xmlns:soap="">
                    <soap:item>Cisco RIS Data Collector PerfMonLog</soap:item>

That looks wrong to me. It should be

input {
    http_poller {
        urls => {
            randomName => {
                method => post
                url => [...]

Thanks @Badger, the syntax was broken having to do with the urls => and then nesting specific urls with "randomName" below them.

Now I have another problem, it seems as though the HTTPs request is failing due to the self-signed certificates:

    "http_request_failure" => {
              "backtrace" => nil,
                  "error" => "PKIX path building failed: unable to find valid certification path to requested target",

From the documentation, it looks like I have to create a JKS and download the certificate in advance. How does this actually work with Logstash in the official Docker containers? Having to manually download certificates and manage a Java Key Store with passwords is not really easy to scale/automate? Is there some way to automatically accept the certificate?

OK, it looks like there is no easy way to do this as the following "ignore" switches were depreciated.

ssl => true
ssl_certificate_verification => false

There are several outstanding issues on http_poller input. Is there anyway we can get the "ignore" switch back on to handle trusted SSL/TLS HTTPs endpoints? What can we do to get this back in the input?

I had issues with certificates when using the http_poller input. I kept getting SSL Errors for handshake alert: unrecognized_name.

I ended up having to put:

-Djsse.enableSNIExtension=false in the /etc/logstash/jvm.options

After I restarted logstash, I was up and running.

@raged Thanks for this hint, but unfortunately it isn't working. I passed in the switch using the LS_JAVA_OPT environment parameter, and I can see it in the Docker container, but I get this in the logs:

jcmd 1 VM.system_properties
#Mon Dec 09 22:02:30 UTC 2019
 "error" => "PKIX path building failed: unable to find valid certification path to requested target",

Maybe I'm still doing something wrong? I tried to find another switch to skip certificate checking (as this is just a lab/test server with self-signed certificate) but didn't find anything supported by java.


Please any help.
I have soap http logstash output and i would like to post some grok fields output using soap webservice.
I used the logstash sprintf syntax ${fieldname} but i doesn't work.(error message: cannot evaluate environnement variable)
Any help please

Thnx in advance

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.