# How to send data to a http end point using LS output plugin **URL:** \ **Category:** Logstash\ **Created:** [August 25, 2016, 4:43pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959 "2016-08-25T16:43:29Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 25, 2016, 4:43pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/1 "2016-08-25T16:43:29Z")
Hi All I have a CSV file which contains three simple fields and i want post its value one after another to http endpoint using LS http output plugin The csv files contain 3 header : Name, Description, Age The end point WSDL URL is `https://abc.xyz.com/acnUserService/UserService?WSDL` I have its username and Password For End point i have a XML ``` $(Description) $(Name) $(Age) ``` Now am trying to use input plugin as file and output plugin as http out plugin. For me realty is i dont know how to use http output effectively i have tried this in conf file ``` input { file { sincedb_path=> "E:/abcde/sincedb_path.txt" path => "E:/logstash-all-plugins-2.3.4/logstash/bin/user.csv" type => "user" start_position => "beginning" } } filter { csv { columns => ["Name","Description","Age"] separator => "," } } output { http { url => "https://abc.xyz.com/acnUserService/UserService?WSDL" http_method => "post" format => "form" mapping => ["Name", "%{Name}", "Description","%{Description}", "Age", "%{Age}"] headers => ["username","password"] } } ``` ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 26, 2016, 3:24am UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/2 "2016-08-26T03:24:29Z")
Did any one tried this similar? ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 26, 2016, 2:33pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/3 "2016-08-26T14:33:37Z")
i have tried this again, but no luck. Please let me know if anyone else has tried this kind of solution before ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 29, 2016, 12:33pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/4 "2016-08-29T12:33:30Z")
Can anyone suggest me example conf file for SOAP Endpoint URL with http output plugin or any other plugin is available to do this activity? ---
**Author:** ![wiibaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiibaa/32/44931_2.png) [@wiibaa](https://discuss.elastic.co/u/wiibaa)\ **Post date:** [August 30, 2016, 9:41am UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/5 "2016-08-30T09:41:29Z")
Hi @rkhapre, sorry for the frustration with using this plugin. It is true that http plugin is too low-level for SOAP calls but as not specific plugin is available for that it should still be possible with perseverance. Sorry that I cannot test it further but this should hopefully drive you towards a solution. - there is a specific config for the content\_type - headers is a **hash** meaning there is a better way to write in the configuration, see below - this plugin does not support basic auth as-is, you need to supply the Authorization header manually - the **form** config is not what you want, writing a SOAP message require the **message** format with the **message** config, I do not recall how to write multiline string in config, so you may preferably inline your XML - to use event fields in the message you must use the logstash **sprintf** syntax **${fieldname}** , in case of nested fields (objects) the syntax is **${"[outer][inner]"}** That should look like: ``` output { http { url => "https://xyz.com:443/acnuserService/userService" http_method => "post" content_type => "text/xml;charset=UTF-8" headers => { "SOAPAction" => "http://xmlns.xyz.com/apps/abc/sdfg/user/userService/createuser" "Authorization" => "Basic a2V2aW4uc2Nob3R0OlhEQzg4NTk0" } format => "message" message => ' ${Description}${Name}${Age}' } } ``` HTH ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 30, 2016, 1:06pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/6 "2016-08-30T13:06:36Z")
Hi @wiibaa First of all thanks a lot for replying me on this post. Really appreciate for this I tried what you have suggested here , but no luck. For URL i want to know information that shall i use WSDL URL or endpoint url? I tried with both, but now i am not getting any error message in the log file using --debug After a certain step the logstash closes on its own So basically its not fitting the purpose Below is the header which work perfectly in SOAP UI ``` POST https://xyz.com:443/acnuserService/userService HTTP/1.1 Accept-Encoding: gzip,deflate Content-Type: text/xml;charset=UTF-8 SOAPAction: "http://xmlns.xyz.com/apps/abc/sdfg/user/userService/createuser" Content-Length: 625 Host: xyz.com:443 Connection: Keep-Alive User-Agent: Apache-HttpClient/4.1.1 (java 1.5) Authorization: Basic a2V2aW4uc2Nob3R0OlhEQzg4NTk0 ``` Is it possible to use old logstash plugin with new logstash version I found one plugin called logstash-soap which works on savon ruby > **[stockholmuniversity/logstash-soap](https://github.com/stockholmuniversity/logstash-soap)** > > logstash-soap - A Logstash SOAP output plugin Any thoughts on to install with new LS? ---
**Author:** ![wiibaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiibaa/32/44931_2.png) [@wiibaa](https://discuss.elastic.co/u/wiibaa)\ **Post date:** [August 30, 2016, 1:15pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/7 "2016-08-30T13:15:19Z")
The plugin you points to is unusable, it should be reworked to fit logstash plugin architecture first. What version of logstash are you using ? From your soap-ui example I added the URL parameter in my previous post, can you be more specific about "_I tried what you have suggested here , but no luck._" In debug mode you should see logs from the http output plugin for success or failure normally ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 30, 2016, 2:00pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/8 "2016-08-30T14:00:22Z")
Hi in --debug mode i am getting this below message actually its not giving any error. After reading output parameter it give this below messages which is constant and endless ``` {:timestamp=>"2016-08-30T19:21:05.295000+0530", :message=>"Pipeline main started", :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb", :line=>"473", :method=>"start_pipeline"} {:timestamp=>"2016-08-30T19:21:10.303000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"} {:timestamp=>"2016-08-30T19:21:15.310000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"} {:timestamp=>"2016-08-30T19:21:20.349000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"} {:timestamp=>"2016-08-30T19:21:25.357000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"} {:timestamp=>"2016-08-30T19:21:30.364000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"} {:timestamp=>"2016-08-30T19:21:35.372000+0530", :message=>"Pushing flush onto pipeline", :level=>:debug, :file=>"/logstash-all-plugins-2.3.4/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=>"458", :method=>"flush"} ``` ---
**Author:** ![wiibaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiibaa/32/44931_2.png) [@wiibaa](https://discuss.elastic.co/u/wiibaa)\ **Post date:** [August 30, 2016, 2:01pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/9 "2016-08-30T14:01:22Z")
Are you sure events are reaching your output ? ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 30, 2016, 2:28pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/10 "2016-08-30T14:28:38Z")
Hi Yes i have checked multiple times object is not getting created in system Though with SOAP UI it works I am not getting error very strange, i get only those which i have pasted above Before you gave me way to put header, i used to get the response like this Encountered non-200 HTTP code Now i am not getting any response also, not even success response ---
**Author:** ![wiibaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiibaa/32/44931_2.png) [@wiibaa](https://discuss.elastic.co/u/wiibaa)\ **Post date:** [August 30, 2016, 2:33pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/11 "2016-08-30T14:33:51Z")
It is true that request that are responded with a 200 will not produce any log. Can you try using a HTTP sniffer like wireshark or Fiddler to check what are the request/response that are send/received by logstash, it seems the only way ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 30, 2016, 5:36pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/12 "2016-08-30T17:36:56Z")
I tried this again after re writing the conf file from scratch which is think more concrete based on your inputs now i am getting this error `{:timestamp=>"2016-08-30T23:00:43.473000+0530", :message=>"[HTTP Output Failure] Encountered non-200 HTTP code 200", :response_code=>500, :url=>"https://xyz.com:443/acnUsersService/UserService", :event=>#, @cancelled=false, @data={"message"=>"New low cost houses for middle-class family.\nHouse according to average income salary\n", "@version"=>"1", "@timestamp"=>"2016-08-28T16:30:14.000Z", "received"=>"by 10.176.81.69 with HTTP; Sun, 28 Aug 2016 09:30:14 -0700", "date"=>"Sun, 28 Aug 2016 22:00:14 +0530", "from"=>"User Lake ", "to"=>"User Lake ", "message-id"=>"", "subject"=>"New Houses for Middle Classes", "mime-version"=>"1.0", "content-type"=>"multipart/alternative; boundary=001a114f37ea2401b5053b244355", "delivered-to"=>"sdfg@zzz.com"}, @metadata={}, @accessors=#"New low cost houses for middle-class family.\nHouse according to average income salary\n", "@version"=>"1", "@timestamp"=>"2016-08-28T16:30:14.000Z", "received"=>"by 10.176.81.69 with HTTP; Sun, 28 Aug 2016 09:30:14 -0700", "date"=>"Sun, 28 Aug 2016 22:00:14 +0530", "from"=>"User Lake ", "to"=>"User Lake ",` ---
**Author:** ![wiibaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiibaa/32/44931_2.png) [@wiibaa](https://discuss.elastic.co/u/wiibaa)\ **Post date:** [August 31, 2016, 4:10am UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/13 "2016-08-31T04:10:45Z")
Sadly the plugin logging is incomplete and will not give you the Soap Fault that is in the response body... Do you have access to the SOAP server logs, otherwise you need Wireshark to know what is the content of this response On the other hand, do you have several config files in your running logstash, because this event is clearly an event created by the imap input, and your config above only talk about reading a csv file. So I suspect you have 2 differents config files. You must understand that all config files are concatenated into a single one to create a single pipeline, so all inputs sends to the single filtering pipeline and then go to all outputs. Unless you specify conditions with **if** statements, is it the case ? ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 31, 2016, 4:14am UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/14 "2016-08-31T04:14:31Z")
Hey @wiibaa Want to share a good news with you, i have cracked SOAP with http output plugin. Thanks to you, who asked me to work on HTTP sniffer, that has helped me to crack this I have used same stuff what you told above. The reason i was not getting is the **message** command with single line xml and whitespaces means it will not work I used Notepad++ to trim trailing white spaces and Join line feature of Notepad++ to make it perfect. and used HTTP sniffer to monitor where it is going wrong I think you can claim proudly that http output plugin is meant for SOAP Thanks to you. Big hug Cheers! ---
**Author:** ![wiibaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiibaa/32/44931_2.png) [@wiibaa](https://discuss.elastic.co/u/wiibaa)\ **Post date:** [August 31, 2016, 4:18am UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/15 "2016-08-31T04:18:50Z")
Happy for you 🙂 I will soon register a github issue with documentation enhancement from what we discuss, can you please close the issues you opened there. Thanks ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [August 31, 2016, 9:15am UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/16 "2016-08-31T09:15:03Z")
I have Closed the issue in github. Thanks! ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [September 6, 2016, 3:46am UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/17 "2016-09-06T03:46:53Z")
Hi @wiibaa. If required i can start a new thread. But this is something related Last week i tried csv--\> HTTP Output This time i am trying get information from HTTP End Point --\> Post to Elastic search For this i tried "http input plug" with this i am not able to define URL and message Second option i tried is "http\_poller input plugin" with this i am getting bad response and ES index fields are all related to website statistics Please let me know if you have any idea how to achieve reverse wherein output will beElastic index and input will be the Endpoint SOAP url ---
**Author:** ![wiibaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiibaa/32/44931_2.png) [@wiibaa](https://discuss.elastic.co/u/wiibaa)\ **Post date:** [September 6, 2016, 7:03pm UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/18 "2016-09-06T19:03:30Z")
Hi again @rkhapre, can you please tell me more about your idea. Do you want to **1 =\>** Call logstash as it was a SOAP endpoint by sending it a XML message to the endpoint defined by the http plugin host/port config or **2 =\>** Have logstash call a SOAP endpoint periodically to retrieve data with http\_poller Can you then provide an example of message, XML I suppose, that need to be processed in Logstash ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [September 7, 2016, 2:54am UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/19 "2016-09-07T02:54:45Z")
hi @wiibaa Here is the scenario that i am trying 1. In my previous post, my input was csv file wherein i had 3 fields Name, Description, Age 2. After using SOAP end point with logstash http output plugin, i successfully posted the 3 fields 3. Now i want to retrieve same 3 fields from application and post it to Elastic Search for analysis For this also i am planning to use SOAP Request. I have a endpoint url and findservice. In Logstash my input will be HTTP input OR http\_poller and output will be Elasticsearch index It is working well in SOAP UI. Below are the headers in SOAP U that i am gettingI, i am using find service to retrive the list of users ``` POST https://xyz.com:443/acnuserService/userService HTTP/1.1 Accept-Encoding: gzip,deflate Content-Type: text/xml;charset=UTF-8 SOAPAction: "http://xmlns.xyz.com/apps/abc/sdfg/user/userService/finduserFinduserByName" Content-Length: 2110 Host: xyz.com:443 Connection: Keep-Alive User-Agent: Apache-HttpClient/4.1.1 (java 1.5) Authorization: Basic a2V2aW4uc2Nob3R0OmpKZDQ0NTc3 ``` Also i have a XML file like the one we had above for create user and we used that one in message=\>. This list of user service i want to see in Kibana ---
**Author:** ![rkhapre](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rkhapre/32/48333_2.png) [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Post date:** [September 8, 2016, 10:12am UTC](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959/20 "2016-09-08T10:12:55Z")
Hi @wiibaa Just looking forward for some tips from you, if this is possible with http input/ http\_poller or i will have to look on other plugins [Next page](https://discuss.elastic.co/t/how-to-send-data-to-a-http-end-point-using-ls-output-plugin/58959.md?page=2)