Hello,
I'm trying to poll SNMP counters ( from Meraki ) using logstash SNMP input module. Sometimes the input plugin gets stuck for no apparent reason. Eventually the logs show logstash attempting a shutdown, but it never completes:
[2020-07-09T13:08:09,050][ERROR][logstash.inputs.snmp ][main][test1-snmp] error invoking walk operation on OID: 1.3.6.1.4.1.29671.1.1.5.1.7, ignoring {:exception=>#<LogStash::SnmpClientError: error sending snmp walk request to target 209.206.48.34/16100: Request timed out.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.2/lib/logstash/inputs/snmp/base_client.rb:81:in `block in walk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.2/lib/logstash/inputs/snmp/base_client.rb:76:in `walk'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.2/lib/logstash/inputs/snmp.rb:179:in `block in run'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.2/lib/logstash/inputs/snmp.rb:177:in `block in run'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.2/lib/logstash/inputs/snmp.rb:167:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:345:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:336:in `block in start_input'"]}
[2020-07-09T13:10:27,526][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2020-07-09T13:10:32,676][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>36, "name"=>"[main]<snmp", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.2/lib/logstash/inputs/snmp/base_client.rb:73:in `walk'"}, {"thread_id"=>31, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}, {"thread_id"=>32, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}, {"thread_id"=>33, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}, {"thread_id"=>34, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}]}}
[2020-07-09T13:10:32,683][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[2020-07-09T13:10:37,700][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>36, "name"=>"[main]<snmp", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.2/lib/logstash/inputs/snmp/base_client.rb:73:in `walk'"}, {"thread_id"=>31, "name"=>"[main]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}, {"thread_id"=>32, "name"=>"[main]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}, {"thread_id"=>33, "name"=>"[main]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}, {"thread_id"=>34, "name"=>"[main]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:278:in `block in start_workers'"}]}}
I apologize if i'ts the wrong location to post, I'm new to ELK. Using logstash 7.8 and logstash-input-snmp.
Logstash is running on ubuntu 18.04 with Open JDK 11.
Input configuration:
snmp {
walk => ["1.3.6.1.4.1.29671.1.1.5.1.6", "1.3.6.1.4.1.29671.1.1.5.1.7" ]
hosts => [{host => "udp:snmp.meraki.com/16100" version => "3" timeout => 30000 retries => 3}]
auth_pass => "xxx"
auth_protocol => "sha"
priv_pass => "xxx"
priv_protocol => "aes"
security_name => "xxx"
security_level => "authPriv"
oid_root_skip => 11
id => "test1-snmp"
tags => "test1 snmp"
interval => 300
type => "test1-snmp"
}