Logstash 6.5.1 stops processing immediately after throwing the following error and becomes unresponsive till it is restarted,
[2019-02-27T19:32:19,054][WARN ][logstash.filters.dns ] DNS: timeout on resolving address. {:field=>"resolvedip", :value=>"36.73.146.220"}
[2019-02-27T19:48:05,232][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"Operation not permitted", "backtrace"=>["org/jruby/ext/socket . . . .
[2019-02-27T19:48:05,390][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<IOError: Operation not permitted>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:339:in `send'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:846:in `send'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:680:in `request'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:536:in `block in fetch_resource'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1108:in `block in resolv'", "org/jruby/RubyArray.java:1734:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1106:in `block in resolv'", "org/jruby/RubyArray.java:1734:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1105:in `block in resolv'", "org/jruby/RubyArray.java:1734:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:1103:in `resolv'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:527:in `fetch_resource'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:517:in `each_resource'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:471:in `each_name'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:154:in `block in each_name'", "org/jruby/RubyArray.java:1734:in `each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:153:in `each_name'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/resolv.rb:135:in `getname'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.11/lib/logstash/filters/dns.rb:294:in `getname'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.11/lib/logstash/filters/dns.rb:281:in `block in retriable_getname'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.11/lib/logstash/filters/dns.rb:266:in `block in retriable_request'", "org/jruby/ext/timeout/Timeout.java:117:in `timeout'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.11/lib/logstash/filters/dns.rb:265:in `retriable_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.11/lib/logstash/filters/dns.rb:280:in `retriable_getname'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.11/lib/logstash/filters/dns.rb:225:in `block in reverse'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.11/lib/logstash/filters/dns.rb:192:in `reverse'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-dns-3.0.11/lib/logstash/filters/dns.rb:102:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:162:in `block in multi_filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:44:in `multi_filter'", "(eval):3749:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):3745:in `block in initialize'", "(eval):3766:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):3763:in `block in initialize'", "(eval):973:in `block in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:320:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:286:in `block in start_workers'"]}
[2019-02-27T19:48:05,564][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
It looks to be a similar issue mentioned here (https://github.com/logstash-plugins/logstash-filter-dns/issues/40), where dns filter seems to be causing the problem. Similarity between the bug reported and the issue that i'm facing is, it keeps resolving the IP addresses and stops processing any more event after throwing the aforementioned error.
When tracking it down, it looks like the dns resolve timeout issue is fixed in logstash-filter-dns-3.0.11 plugin (https://github.com/logstash-plugins/logstash-filter-dns/pull/45/files) but i could still see the error is coming from the same 3.0.11 version.
bin/logstash-plugin list --verbose | grep filter-dns
logstash-filter-dns (3.0.11)
I'm not really sure anything else is causing the problem. Please let me know if any other information is required and redirect me in the right direction.