# Sqs output errors cause blocked input pipelines **URL:** \ **Category:** Logstash\ **Created:** [June 14, 2016, 1:17pm UTC](https://discuss.elastic.co/t/sqs-output-errors-cause-blocked-input-pipelines/52741 "2016-06-14T13:17:26Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![NoumanSaleem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noumansaleem/32/9282_2.png) [@NoumanSaleem](https://discuss.elastic.co/u/NoumanSaleem)\ **Post date:** [June 14, 2016, 1:17pm UTC](https://discuss.elastic.co/t/sqs-output-errors-cause-blocked-input-pipelines/52741/1 "2016-06-14T13:17:26Z")
It's become a daily occurrence in the last week where we see a drop in logs in Elasticsearch caused by blocked pipelines in our logstash stack. We utilize a cluster of logstash for accepting and pushing logs into an SQS queue, and another cluster reading from the queue, filtering, and pushing to elasticsearch. This particular error occurs on the first cluster, and is only resolved by restarting the logstash process (container). ```auto {:timestamp=>"2016-06-14T13:04:34.272000+0000", :message=>"Failed to flush outgoing items", :outgoing_count=>5, :exception=>"AWS::Errors::Base", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in `return_or_raise'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:in `client_request'", "(eval):3:in `send_message_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/sqs/queue.rb:551:in `batch_send'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-sqs-2.0.4/lib/logstash/outputs/sqs.rb:129:in `flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:193:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:in `buffer_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-sqs-2.0.4/lib/logstash/outputs/sqs.rb:121:in `receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/outputs/base.rb:83:in `multi_receive'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/outputs/base.rb:83:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/output_delegator.rb:129:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:warn} {:timestamp=>"2016-06-14T13:04:53.142000+0000", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn} {:timestamp=>"2016-06-14T13:04:53.534000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} ``` the sqs output conf: ```auto output { sqs { batch_events => 5 queue => "${SQS_OUTPUT_QUEUE}" region => "${AWS_REGION}" } } ``` the beats input conf: ```auto input { beats { port => 5044 ssl => true ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder/lumberjack.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder/lumberjack.key" } } ``` These nodes do not perform any filtering, just input -\> queue. The Dockerfile: ```auto FROM logstash:2.3.1 ENV SERVICE_NAME=logstash CMD ["--allow-env", "-f", "/opt/config"] COPY ./config/shipper /opt/config ``` Unfortunately, the error is not very helpful. I am assuming it is `BatchRequestTooLong`, but that is just a guess. For now, I will disable batch sending. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2016, 10:17am UTC](https://discuss.elastic.co/t/sqs-output-errors-cause-blocked-input-pipelines/52741/2 "2016-06-19T10:17:25Z")
There's nothing in the ES logs? ---
**Author:** ![NoumanSaleem](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/noumansaleem/32/9282_2.png) [@NoumanSaleem](https://discuss.elastic.co/u/NoumanSaleem)\ **Post date:** [June 20, 2016, 1:30pm UTC](https://discuss.elastic.co/t/sqs-output-errors-cause-blocked-input-pipelines/52741/3 "2016-06-20T13:30:34Z")
@magnusbaeck our setup resembles the last diagram on the Deploying Scaling Logstash guide [https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html](https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html) The issue occurs in the first set of Logstash instances responsible for outputing logs to the queue (SQS), and because of that we do not receive any logs in ES. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:51am UTC](https://discuss.elastic.co/t/sqs-output-errors-cause-blocked-input-pipelines/52741/4 "2017-07-06T04:51:52Z")