How to send data to a http end point using LS output plugin

rkhapre · August 25, 2016, 4:43pm

Hi All

I have a CSV file which contains three simple fields and i want post its value one after another to http endpoint using LS http output plugin

The csv files contain 3 header : Name, Description, Age

The end point WSDL URL is

https://abc.xyz.com/acnUserService/UserService?WSDL

I have its username and Password

For End point i have a XML

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ide="http://xmlns.xxxx.com/apps/yyy" xmlns:typ="http://xmlns.xxxx.com/apps/yyyy/abc/types/">
   <soapenv:Header/>
   <soapenv:Body>
      <typ:createuser>
         <typ:user>
            <ide:Description>$(Description)</ide:Description>
            <ide:Name>$(Name)</ide:Name>
            <ide:age>$(Age)</ide:age>
         </typ:user>
      </typ:createuser>
   </soapenv:Body>
</soapenv:Envelope>

Now am trying to use input plugin as file and output plugin as http out plugin.
For me realty is i dont know how to use http output effectively

i have tried this in conf file

    input {  
          file {
    	  sincedb_path=>  "E:/abcde/sincedb_path.txt"
              path => "E:/logstash-all-plugins-2.3.4/logstash/bin/user.csv"
              type => "user"
              start_position => "beginning"
          }
    }

filter {  
    csv {
        columns => ["Name","Description","Age"]
        separator => ","
    }
}

    output {
      http {
        url => "https://abc.xyz.com/acnUserService/UserService?WSDL"
        http_method => "post"
        format => "form"
        mapping => [ "Name", "%{Name}", "Description","%{Description}", "Age", "%{Age}" ]
    	headers => ["username","password"]
      }
    }

rkhapre · August 26, 2016, 3:24am

Did any one tried this similar?

rkhapre · August 26, 2016, 2:33pm

i have tried this again, but no luck.

Please let me know if anyone else has tried this kind of solution before

rkhapre · August 29, 2016, 12:33pm

Can anyone suggest me example conf file for SOAP Endpoint URL with http output plugin or any other plugin is available to do this activity?

wiibaa · August 30, 2016, 9:41am

Hi @rkhapre, sorry for the frustration with using this plugin.
It is true that http plugin is too low-level for SOAP calls but as not specific plugin is available for that it should still be possible with perseverance.

Sorry that I cannot test it further but this should hopefully drive you towards a solution.

there is a specific config for the content_type
headers is a hash meaning there is a better way to write in the configuration, see below
this plugin does not support basic auth as-is, you need to supply the Authorization header manually
the form config is not what you want, writing a SOAP message require the message format with the message config, I do not recall how to write multiline string in config, so you may preferably inline your XML
to use event fields in the message you must use the logstash sprintf syntax ${fieldname}, in case of nested fields (objects) the syntax is ${"[outer][inner]"}

That should look like:

output {
    http {
         url => "https://xyz.com:443/acnuserService/userService"
         http_method => "post"
         content_type => "text/xml;charset=UTF-8"
         headers => {
           "SOAPAction" => "http://xmlns.xyz.com/apps/abc/sdfg/user/userService/createuser"
           "Authorization" => "Basic a2V2aW4uc2Nob3R0OlhEQzg4NTk0"
         }
         format => "message"
         message => '<?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ide="http://xmlns.xxxx.com/apps/yyy" xmlns:typ="http://xmlns.xxxx.com/apps/yyyy/abc/types/"> <soapenv:Header/><soapenv:Body> <typ:createuser><typ:user><ide:Description>${Description}</ide:Description><ide:Name>${Name}</ide:Name><ide:age>${Age}</ide:age></typ:user></typ:createuser></soapenv:Body></soapenv:Envelope>'
    }
}

HTH

rkhapre · August 30, 2016, 1:06pm

Hi @wiibaa

First of all thanks a lot for replying me on this post. Really appreciate for this

I tried what you have suggested here , but no luck.
For URL i want to know information that shall i use WSDL URL or endpoint url?
I tried with both, but now i am not getting any error message in the log file using --debug
After a certain step the logstash closes on its own

So basically its not fitting the purpose

Below is the header which work perfectly in SOAP UI

POST https://xyz.com:443/acnuserService/userService HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
SOAPAction: "http://xmlns.xyz.com/apps/abc/sdfg/user/userService/createuser"
Content-Length: 625
Host: xyz.com:443
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
Authorization: Basic a2V2aW4uc2Nob3R0OlhEQzg4NTk0

Is it possible to use old logstash plugin with new logstash version
I found one plugin called logstash-soap which works on savon ruby

Any thoughts on to install with new LS?

wiibaa · August 30, 2016, 1:15pm

The plugin you points to is unusable, it should be reworked to fit logstash plugin architecture first.
What version of logstash are you using ?

From your soap-ui example I added the URL parameter in my previous post, can you be more specific about "I tried what you have suggested here , but no luck." In debug mode you should see logs from the http output plugin for success or failure normally

rkhapre · August 30, 2016, 2:00pm

Hi

in --debug mode i am getting this below message

actually its not giving any error. After reading output parameter it give this below messages which is constant and endless

{:timestamp=>"2016-08-30T19:21:05.295000+0530", :message=>"Pipeline main started", :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb", :line=>"473", :method=>"start_pipeline"}
{:timestamp=>"2016-08-30T19:21:10.303000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}
{:timestamp=>"2016-08-30T19:21:15.310000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}
{:timestamp=>"2016-08-30T19:21:20.349000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}
{:timestamp=>"2016-08-30T19:21:25.357000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}
{:timestamp=>"2016-08-30T19:21:30.364000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}
{:timestamp=>"2016-08-30T19:21:35.372000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"}

wiibaa · August 30, 2016, 2:01pm

Are you sure events are reaching your output ?

rkhapre · August 30, 2016, 2:28pm

Hi

Yes i have checked multiple times object is not getting created in system
Though with SOAP UI it works

I am not getting error very strange, i get only those which i have pasted above

Before you gave me way to put header, i used to get the response like this
Encountered non-200 HTTP code

Now i am not getting any response also, not even success response

wiibaa · August 30, 2016, 2:33pm

It is true that request that are responded with a 200 will not produce any log.
Can you try using a HTTP sniffer like wireshark or Fiddler to check what are the request/response that are send/received by logstash, it seems the only way

rkhapre · August 30, 2016, 5:36pm

I tried this again after re writing the conf file from scratch
which is think more concrete based on your inputs

now i am getting this error

{:timestamp=>"2016-08-30T23:00:43.473000+0530", :message=>"[HTTP Output Failure] Encountered non-200 HTTP code 200", :response_code=>500, :url=>"https://xyz.com:443/acnUsersService/UserService", :event=>#<LogStash::Event:0x11992019 @metadata_accessors=#<LogStash::Util::Accessors:0x663dcd5b @store={}, @lut={}>, @cancelled=false, @data={"message"=>"New low cost houses for middle-class family.\nHouse according to average income salary\n", "@version"=>"1", "@timestamp"=>"2016-08-28T16:30:14.000Z", "received"=>"by 10.176.81.69 with HTTP; Sun, 28 Aug 2016 09:30:14 -0700", "date"=>"Sun, 28 Aug 2016 22:00:14 +0530", "from"=>"User Lake <Userlake@zzz.com>", "to"=>"User Lake <sdfg@zzz.com>", "message-id"=>"<CAHDMMzAXSmScA-L8qD+MTBA4MDO9YNTD+c+1jX=07xEx2GjWoA@mail.zzz.com>", "subject"=>"New Houses for Middle Classes", "mime-version"=>"1.0", "content-type"=>"multipart/alternative; boundary=001a114f37ea2401b5053b244355", "delivered-to"=>"sdfg@zzz.com"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x3a1506a9 @store={"message"=>"New low cost houses for middle-class family.\nHouse according to average income salary\n", "@version"=>"1", "@timestamp"=>"2016-08-28T16:30:14.000Z", "received"=>"by 10.176.81.69 with HTTP; Sun, 28 Aug 2016 09:30:14 -0700", "date"=>"Sun, 28 Aug 2016 22:00:14 +0530", "from"=>"User Lake <sdfg@zzz.com>", "to"=>"User Lake <sdfg@zzz.com>",

wiibaa · August 31, 2016, 4:10am

Sadly the plugin logging is incomplete and will not give you the Soap Fault that is in the response body...
Do you have access to the SOAP server logs, otherwise you need Wireshark to know what is the content of this response

On the other hand, do you have several config files in your running logstash, because this event is clearly an event created by the imap input, and your config above only talk about reading a csv file.
So I suspect you have 2 differents config files. You must understand that all config files are concatenated into a single one to create a single pipeline, so all inputs sends to the single filtering pipeline and then go to all outputs. Unless you specify conditions with if statements, is it the case ?

rkhapre · August 31, 2016, 4:14am

Hey @wiibaa

Want to share a good news with you, i have cracked SOAP with http output plugin.
Thanks to you, who asked me to work on HTTP sniffer, that has helped me to crack this

I have used same stuff what you told above.
The reason i was not getting is the message command with single line xml and whitespaces means it will not work
I used Notepad++ to trim trailing white spaces and Join line feature of Notepad++ to make it perfect.
and used HTTP sniffer to monitor where it is going wrong

I think you can claim proudly that http output plugin is meant for SOAP

Thanks to you.

Big hug

Cheers!

wiibaa · August 31, 2016, 4:18am

Happy for you

I will soon register a github issue with documentation enhancement from what we discuss, can you please close the issues you opened there.
Thanks

rkhapre · August 31, 2016, 9:15am

I have Closed the issue in github.
Thanks!

rkhapre · September 6, 2016, 3:46am

Hi @wiibaa.

If required i can start a new thread. But this is something related
Last week i tried csv--> HTTP Output

This time i am trying get information from HTTP End Point --> Post to Elastic search

For this i tried "http input plug" with this i am not able to define URL and message

Second option i tried is "http_poller input plugin" with this i am getting bad response and ES index fields are all related to website statistics

Please let me know if you have any idea how to achieve reverse wherein output will beElastic index
and input will be the Endpoint SOAP url

wiibaa · September 6, 2016, 7:03pm

Hi again @rkhapre,

can you please tell me more about your idea. Do you want to

1 => Call logstash as it was a SOAP endpoint by sending it a XML message to the endpoint defined by the http plugin host/port config

or

2 => Have logstash call a SOAP endpoint periodically to retrieve data with http_poller

Can you then provide an example of message, XML I suppose, that need to be processed in Logstash

rkhapre · September 7, 2016, 2:54am

hi @wiibaa

Here is the scenario that i am trying

In my previous post, my input was csv file wherein i had 3 fields Name, Description, Age
After using SOAP end point with logstash http output plugin, i successfully posted the 3 fields
Now i want to retrieve same 3 fields from application and post it to Elastic Search for analysis

For this also i am planning to use SOAP Request. I have a endpoint url and findservice.
In Logstash my input will be HTTP input OR http_poller and output will be Elasticsearch index

It is working well in SOAP UI.

Below are the headers in SOAP U that i am gettingI, i am using find service to retrive the list of users

POST https://xyz.com:443/acnuserService/userService HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
SOAPAction: "http://xmlns.xyz.com/apps/abc/sdfg/user/userService/finduserFinduserByName"
Content-Length: 2110
Host: xyz.com:443
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
Authorization: Basic a2V2aW4uc2Nob3R0OmpKZDQ0NTc3

Also i have a XML file like the one we had above for create user and we used that one in message=>.

This list of user service i want to see in Kibana

rkhapre · September 8, 2016, 10:12am

Hi @wiibaa

Just looking forward for some tips from you, if this is possible with http input/ http_poller or i will have to look on other plugins