Recently I had an issue where my Elastic Cloud instance was not responding well. Logstash was throwing a lot of errors about not being able to send events, and I noticed that some of my other outputs weren't getting processed, specifically critical exec outputs.
I'm running logstash-5.0.0-alpha5 which was upgraded from 2.3 and the elasticsearch-output version 4.1.3 which I think is 8 or 9 months old.
If one output is blocked is it expected that others will be? That the whole pipeline will be blocked?
I wondering if this is related to outdated and alpha versions of the products or if this behavior is expected, and I should design my systems to handle this.
Here's the log:
{
:timestamp=>"2017-03-12T22:44:35.915000-0700",
:message=>"Encountered an unexpected error submitting a bulk request! Will retry.",
:error_message=>"Cabin::Channel#error(message, data={}) - second argument you gave me wasa Array, I require a Hash.",
:class=>"ArgumentError",
:backtrace=>[
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/cabin-0.8.1/lib/cabin/mixins/logger.rb:72:in `error'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-4.1.3-java/lib/logstash/outputs/elasticsearch/common.rb:75:in `retrying_submit'",
"org/jruby/RubyArray.java:1613:in `each'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-4.1.3-java/lib/logstash/outputs/elasticsearch/common.rb:75:in `retrying_submit'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-4.1.3-java/lib/logstash/outputs/elasticsearch/common.rb:23:in `multi_receive'",
"org/jruby/RubyArray.java:1653:in `each_slice'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-4.1.3-java/lib/logstash/outputs/elasticsearch/common.rb:22:in `multi_receive'",
"/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:136:in `threadsafe_multi_receive'",
"/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:122:in `multi_receive'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:291:in `output_batch'",
"org/jruby/RubyHash.java:1342:in `each'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:291:in `output_batch'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:242:in `worker_loop'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:215:in `start_workers'"
],
:level=>:error
}