Logstash fails to send data to Elasticsearch

[amruth]

Hi,

I have a Logstash running which sends data to Elasticsearch nodes and also to other Logstash over http output plugin. Problem I see is, whenever the Losgtash(destination) is stopped(for whatever reason), Logstash(source) fails to send data to Elasticsearch as well. Here is the part of my config,

output{
        if [component] == "Something"{

                file {
                        path => "/tmp/logstash/outputfornow.log"
                }

                elasticsearch {
                        hosts => ["xx.xx.xx:9200","xx.xx.xx:9200","xx.xx.xx:9200"]
                        index => "index-%{+YYYY.MM}"
                }

                http {
                        url=> "http://xx.xx.xx:443"
                        http_method=> "post"
                        format=> "json"
                }

        }

Can someone please say the problem here?

[yaauie (Ry Biesemeyer)]

Outputs are somewhat all-or-nothing.

When each event gets to the outputs block, it occupies one of the worker threads until all matching outputs complete; if one output is very slow to respond, or enters into a retry sequence for the event, that thread becomes blocked until the event can complete all configured outputs.

Presently, this is by design, or rather, because we don't have a better design just yet; if we were to allow arbitrary buffering within the outputs, it would be really easy to create situations where we either have partial failure, or where one output gets so far ahead of the other(s) that we consume all available memory and crash.