Http output plugin is not working in logstash, Please help

Hi,
I am trying to send json message to kibana api using http plugin, I found the below error, Please help and appreciated.

[2021-06-14T14:45:26,361][ERROR][logstash.outputs.http    ] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"https://XXXXXXXXXXXXXXXXXXXXXX:5601/api/saved_objects/_bulk_create", :event=>#<LogStash::Event:0x530a92cd>}

Please fine the below plugin options I used:
-------------------------------------------------------------------------------------
input plugin:
input {
       http {
             port => XXXX
             codec => json_lines
             ssl => true
             ssl_certificate => "XXXXXX"
             ssl_key => "XXXXXX"
       }
}
-------------------------------------------------------------------------------------
output plugin:
output {
    http {
         url => "https://XXXXXXXXXXXXXXXXXXXXXXX:5601/api/saved_objects/_bulk_create"
         http_method => "post"
         codec => "json_lines"
         cacert => "/var/tmp/XXXXXXXXXXX.pem"
         headers => {
           "Authorization" => "Basic XXXXXXXXXXXXXXXXXXXXXX"
           "Content-Type" => "application/json"
           "kbn-xsrf" => "reporting"  
         }
    }
}
-------------------------------------------------------------------------------------

saved_object/_bulk_create expects to get an array of JSON objects. The json_lines codec will encode a single JSON object. _bulk_create will not accept that. You could try something like

filter {
    ruby { code => 'event.set("[@metadata][eventAsJson]", event.to_json)' }
}
output {
    http {
       codec => line { format => "[ %{[@metadata][eventAsJson]} ]" }
        ....

Of course this assumes that your event is a valid object, which seems unlikely, since logstash may have added fields to the event (e.g. @timestamp) that kibana is not expecting.

Hi Badger, Thanks for reply.

I have added the above entries, but still not working.
I dont see any errors in logstash and started successfully.
but I dont see any entries in the log that messages are dropping, I am trying to understand where the messages are dropping.
I am trying to create multiple objects by sending json form of kibana objects to kibana api.
please help and let me know if I am missing anything else.

please let me know how can I drop the fields that are added by logstash.

Thanks

You can remove fields using

mutate { remove_field => [ "someField", "anotherField" ] }

I suggest you replace the http output with a file or stdout output using the same codec and see if the request is a valid kibana request.

Thanks for reply,
I am able to remove those fields.
now I am able to see the message body in the log file and compared with the input json data.
I found one additional field "tags":["_rubyexception"]}" in the body and found the below error.

Please advise.

:message=>"XXXXXXXXXXXXXXXXXXXXXXXXXXX:5601 failed to respond", :class=>"Manticore::ClientProtocolException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in block in initialize'", "/opt/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in call'", "/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:239:in send_event'", "/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:175:in send_events'", "/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-http-5.2.4/lib/logstash/outputs/http.rb:124:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in multi_receive'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:390:in block in output_batch'", "org/jruby/RubyHash.java:1419:in each'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:389:in output_batch'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:341:in worker_loop'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:304:in block in start_workers'"], :will_retry=>true}

If you have a _rubyexception tag then the ruby filter should have logged an exception.

I have no idea why Manticore is logging that 'failed to respond' error.

I have removed the ruby filter and removed the _rubyexception tag.
Now the message body is matching with input json.
but still I am getting 'failed to respond' error.
Please advise if there is any other options I need to try.

Please let me know if anyone facing same kind of issue,
I am able to create saved objects by ingesting the json format into saved_objects Kibana api with curl command. but when I use logstash to do the same job using http output plugin, it is failing.
Appreciated help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.