Hi All,
Last night on of the redis servers disappeared with the result that the redis input plugin stopped working. Now this is a expected behavior. However Looking at my log I found the plugin trying to restart over and over craching with the error "Error: Too many open files - Too many open files" I can only how this happend but my guess would be that it ended up in this state due to all the retries.
On my logtstash instances I have monitoring and I would have been alerted when the logstash stopped which didn't happen as the redis kept trying to restart.
No sure if this is a bug but when the plugin has an error like above, should it not stop logstash altogether?
Here is the full error:
[2018-08-27T07:36:46,877][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Redis host=>"10.80.3.131", key=>"logstash:redis2", data_type=>"list", codec=><LogStash::Codecs::JSON id=>"json_89f85466-3798-400f-a639-2dfe5c779e5f", enable_metric=>true, charset=>"UTF-8">, threads=>50, id=>"4c7c003952a7d99c432e3248e4e10840f1bbfd831b2de16f98339e19d9ed9468", enable_metric=>true, port=>6379, ssl=>false, db=>0, timeout=>5, batch_count=>125>
Error: Too many open files - Too many open files
Exception: Errno::EMFILE
Stack: org/jruby/ext/socket/RubyTCPSocket.java:147:in `initialize'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:39:in `initialize'
org/jruby/RubyIO.java:875:in `new'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:151:in `block in connect'
org/jruby/ext/timeout/Timeout.java:117:in `timeout'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:150:in `connect'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:313:in `connect'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:336:in `establish_connection'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:101:in `block in connect'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:293:in `with_reconnect'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:100:in `connect'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:364:in `ensure_connected'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:221:in `block in process'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:326:in `logging'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:220:in `process'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis/client.rb:120:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:2385:in `block in script'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:58:in `block in synchronize'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/monitor.rb:214:in `mon_synchronize'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:58:in `synchronize'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/redis-3.3.5/lib/redis.rb:2384:in `script'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-redis-3.2.2/lib/logstash/inputs/redis.rb:167:in `load_batch_script'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-redis-3.2.2/lib/logstash/inputs/redis.rb:154:in `connect'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-redis-3.2.2/lib/logstash/inputs/redis.rb:194:in `list_runner'
org/jruby/RubyMethod.java:115:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-redis-3.2.2/lib/logstash/inputs/redis.rb:101:in `run'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:in `block in start_input'