Logstash Crash when SQS plugin fails


(Chris Cundil) #1

The logstash process is exiting when the SQS plugin receives a networking error. It's frustrating enough that there are no specifics provided around this networking error but the fact that it makes logstash fall over makes it infuriating!

Here's the log output at the time of the crash:

$tail -500 /var/log/logstash/logstash.log
{:timestamp=>"2016-11-08T06:03:03.759000+0000", 
:message=>"Pipeline aborted due to error", :exception=>"Seahorse::Client::NetworkingError",
:backtrace=>["org/jruby/ext/openssl/SSLSocket.java:803:in `sysread_nonblock'",
"/opt/logstash/vendor/jruby/lib/ruby/shared/jopenssl19/openssl/buffering.rb:174:in   
`read_nonblock'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:141:in `rbuf_fill'", 
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:122:in `readuntil'", 
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:132:in `readline'", 
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:2571:in `read_status_line'", 
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:2560:in `read_new'", 
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:796:in `connect'", 
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:756:in `do_start'", 
"/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:751:in `start'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/connection_pool.rb:281:in `start_session'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/connection_pool.rb:93:in `session_for'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/handler.rb:116:in `session'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/handler.rb:68:in `transmit'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/net_http/handler.rb:42:in `call'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/content_length.rb:12:in `call'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/request_signer.rb:84:in `call'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:87:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:118:in `retry_request'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:101:in `retry_if_possible'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:89:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:118:in `retry_request'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:101:in `retry_if_possible'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:89:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:118:in `retry_request'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:101:in `retry_if_possible'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/retry_errors.rb:89:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/sqs_queue_urls.rb:13:in `call'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/query/handler.rb:27:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/endpoint.rb:41:in `call'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/response_target.rb:21:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/request.rb:70:in `send_request'", "<<TRUNCATED>>"], :level=>:error}

Note we're using Logstash version 2.4.0

Has anyone else experienced any such problems? Has anyone found a resolution?


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.