Http_poller error when building XML/SOAP request

Elastic Stack Logstash
1

Hi there, I'm trying to use the http_poller input to periodically generate an HTTP/XML/SOAP requests but I'm getting "Invalid URL post" back - Logstash doesn't even seem to be building the request:

Pipeline aborted due to error {:pipeline_id=>"ucm-get-perfmon", :exception=>#<LogStash::ConfigurationError: Invalid URL post>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:103:in `validate_request!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:95:in `normalize_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:55:in `block in setup_requests!'", "org/jruby/RubyHash.java:1417:in `each'", "org/jruby/RubyEnumerable.java:851:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:55:in `setup_requests!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http_poller-5.0.1/lib/logstash/inputs/http_poller.rb:45:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:199:in `block in register_plugins'", "org/jruby/RubyArray.java:1800:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:198:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:303:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:259:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:153:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:108:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipelines/ucm-get-perfmon.conf"], :thread=>"#<Thread:0x7653662d run>"}

My config for logstash looks like this:

input {
    http_poller {
        urls => {
            method => post
            url => "https://server.com:8443/logcollectionservice2/services/LogCollectionPortTypeService?wsdl"
            auth => {
                user => "user"
                password => "password"
            }
            headers => {
                "Content-Type" => "text/xml"
                "SOAPAction" => "CUCM:DB ver=12.5"
            }
            body => '<!--LogCollection API - SelectLogFiles - Request--><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soap="http://schemas.cisco.com/ast/soap"><soapenv:Header/><soapenv:Body><soap:selectLogFiles><soap:FileSelectionCriteria><soap:ServiceLogs><soap:item></soap:item></soap:ServiceLogs><soap:SystemLogs><soap:item>Cisco RIS Data Collector PerfMonLog</soap:item></soap:SystemLogs><soap:SearchStr></soap:SearchStr><soap:Frequency>OnDemand</soap:Frequency><soap:JobType>PushtoSFTPServer</soap:JobType><soap:ToDate></soap:ToDate><soap:FromDate></soap:FromDate><soap:TimeZone></soap:TimeZone><soap:RelText>Days</soap:RelText><soap:RelTime>5</soap:RelTime><soap:Port>22</soap:Port><soap:IPAddress>labserver.com</soap:IPAddress><soap:UserName>username</soap:UserName><soap:Password>password</soap:Password><soap:ZipInfo>false</soap:ZipInfo><soap:RemoteFolder>/data/tracelogger/test/</soap:RemoteFolder></soap:FileSelectionCriteria></soap:selectLogFiles></soapenv:Body></soapenv:Envelope>'
        }
        request_timeout => 60
        # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
        schedule => { cron => "* * * * * UTC" }
    }
}

output {
  stdout {
    codec => rubydebug
  }
}

I can get the request working OK using postman and it looks like this:

POST /logcollectionservice2/services/LogCollectionPortTypeService?wsdl HTTP/1.1
Host: hostname.com:8443
SOAPAction: CUCM:DB ver=12.5 
Content-Type: text/xml
Authorization: Basic dHJhY2Vsb2dnZXIuYXhsLXJvLmFkbWluOnp5bU9pRGVybk1vcg==
User-Agent: PostmanRuntime/7.20.1
Accept: */*
Cache-Control: no-cache
Postman-Token: 65b97d05-1e4f-44cd-af48-935989a062ef,47702e32-3cf2-4668-9a79-8c82e26a5e9f
Host: hostname.com:8443
Accept-Encoding: gzip, deflate
Content-Length: 1422
Cookie: SoapSession.id=88B86D797786C68C381575460218344; JSESSIONID=A474438A26B671FEBE53C0DF917B0A9E; JSESSIONIDSSO=DC7D5E497DC11A2B3EF86107207F278B
Connection: keep-alive
cache-control: no-cache

<!--LogCollection API - SelectLogFiles - Request-->
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soap="http://schemas.cisco.com/ast/soap">
    <soapenv:Header/>
    <soapenv:Body>
        <soap:selectLogFiles>
            <soap:FileSelectionCriteria>
                <soap:ServiceLogs>
                    <soap:item></soap:item>
                </soap:ServiceLogs>
                <soap:SystemLogs>
                    <soap:item>Cisco RIS Data Collector PerfMonLog</soap:item>
                </soap:SystemLogs>
                <soap:SearchStr></soap:SearchStr>
                <soap:Frequency>OnDemand</soap:Frequency>
                <soap:JobType>PushtoSFTPServer</soap:JobType>
                <soap:ToDate></soap:ToDate>
                <soap:FromDate></soap:FromDate>
                <soap:TimeZone></soap:TimeZone>
                <soap:RelText>Days</soap:RelText>
                <soap:RelTime>5</soap:RelTime>
                <soap:Port>22</soap:Port>
                <soap:IPAddress>server.com</soap:IPAddress>
                <soap:UserName>username</soap:UserName>
                <soap:Password>password</soap:Password>
                <soap:ZipInfo>false</soap:ZipInfo>
                <soap:RemoteFolder>/data/tracelogger/test/</soap:RemoteFolder>
            </soap:FileSelectionCriteria>
        </soap:selectLogFiles>
    </soapenv:Body>
</soapenv:Envelope>
2

That looks wrong to me. It should be

input {
    http_poller {
        urls => {
            randomName => {
                method => post
                url => [...]
3

Thanks @Badger, the syntax was broken having to do with the urls => and then nesting specific urls with "randomName" below them.

Now I have another problem, it seems as though the HTTPs request is failing due to the self-signed certificates:

    "http_request_failure" => {
              "backtrace" => nil,
                  "error" => "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",

From the documentation, it looks like I have to create a JKS and download the certificate in advance. How does this actually work with Logstash in the official Docker containers? Having to manually download certificates and manage a Java Key Store with passwords is not really easy to scale/automate? Is there some way to automatically accept the certificate?

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html

4

OK, it looks like there is no easy way to do this as the following "ignore" switches were depreciated.

ssl => true
ssl_certificate_verification => false

There are several outstanding issues on http_poller input. Is there anyway we can get the "ignore" switch back on to handle trusted SSL/TLS HTTPs endpoints? What can we do to get this back in the input?



5

I had issues with certificates when using the http_poller input. I kept getting SSL Errors for handshake alert: unrecognized_name.

I ended up having to put:

-Djsse.enableSNIExtension=false in the /etc/logstash/jvm.options

After I restarted logstash, I was up and running.

6

@raged Thanks for this hint, but unfortunately it isn't working. I passed in the switch using the LS_JAVA_OPT environment parameter, and I can see it in the Docker container, but I get this in the logs:

jcmd 1 VM.system_properties
1:
#Mon Dec 09 22:02:30 UTC 2019
awt.toolkit=sun.awt.X11.XToolkit
ls.log.format=plain
java.specification.version=11
jsse.enableSNIExtension=false

 "error" => "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",

Maybe I'm still doing something wrong? I tried to find another switch to skip certificate checking (as this is just a lab/test server with self-signed certificate) but didn't find anything supported by java.

7

Hello,

Please any help.
I have soap http logstash output and i would like to post some grok fields output using soap webservice.
I used the logstash sprintf syntax ${fieldname} but i doesn't work.(error message: cannot evaluate environnement variable)
Any help please

Thnx in advance

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.