The logstash process is exiting when the SQS plugin receives a networking error. It's frustrating enough that there are no specifics provided around this networking error but the fact that it makes logstash fall over makes it infuriating!
Here's the log output at the time of the crash:
$tail -500 /var/log/logstash/logstash.log
{:timestamp=>"2016-11-08T06:03:03.759000+0000",
:message=>"Pipeline aborted due to error", :exception=>"Seahorse::Client::NetworkingError",
:backtrace=>["org/jruby/ext/openssl/SSLSocket.java:803:in `sysread_nonblock'",
"/opt/logstash/vendor/jruby/lib/ruby/shared/jopenssl19/openssl/buffering.rb:174:in
`read_nonblock'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:141:in `rbuf_fill'",
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:122:in `readuntil'",
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:132:in `readline'",
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:2571:in `read_status_line'",
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:2560:in `read_new'",
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:796:in `connect'",
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:756:in `do_start'",
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:751:in `start'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/connection_pool.rb:281:in `start_session'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/connection_pool.rb:93:in `session_for'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/handler.rb:116:in `session'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/handler.rb:68:in `transmit'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/handler.rb:42:in `call'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/content_length.rb:12:in `call'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/request_signer.rb:84:in `call'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:87:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:118:in `retry_request'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:101:in `retry_if_possible'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:89:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:118:in `retry_request'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:101:in `retry_if_possible'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:89:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:118:in `retry_request'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:101:in `retry_if_possible'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:89:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/sqs_queue_urls.rb:13:in `call'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/query/handler.rb:27:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/endpoint.rb:41:in `call'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/response_target.rb:21:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/request.rb:70:in `send_request'", "<<TRUNCATED>>"], :level=>:error}
Note we're using Logstash version 2.4.0
Has anyone else experienced any such problems? Has anyone found a resolution?