SNMP timeout when run from a docker container

Elastic Stack Logstash
1

I'm using Logstash 7.15.1 on both Centos 7.7 and a docker container with the same configuration (below).

When running from Centos, everything works as expected, but on a docker container (in Kubernetes), the SNMP plugin works with the SNMP walk, but errors on the tables config (to the same host!). It doesn't really make sense to me, as I can make the timeouts really high, but it makes no difference at all. I'm not really sure how to go about solving this and any help or suggestions would be appreciated.

Config:

input {
      snmp {
        # Basic Uptime
        # 1.3.6.1.2.1.1.1", # Description
        # 1.3.6.1.2.1.1.2", # Object ID 
        # 1.3.6.1.2.1.1.3", # Uptime
        # 1.3.6.1.2.1.1.4", # Contact
        # 1.3.6.1.2.1.1.5"  # Hostname
        # 1.3.6.1.2.1.1.6"  # Location
        # 1.3.6.1.2.1.1.7"  # Services
        
        walk => ["1.3.6.1.2.1.1"]
        hosts => [
          { host => "udp:172.16.1.4/161" community => "public" }
        ]
        oid_root_skip => 7
        interval => 30
        tags => [
          "snmp"
        ]
      }
      snmp {
        # Network Interfaces
        tables => [ { name => networkinterface "columns" => [
            "1.3.6.1.2.1.2.2.1.1",    # - Inteface index
            "1.3.6.1.2.1.2.2.1.2",    # - Description
            "1.3.6.1.2.1.2.2.1.4",    # - MTU
            "1.3.6.1.2.1.2.2.1.5",    # - Speed
            "1.3.6.1.2.1.2.2.1.6",    # - Physical Address
            "1.3.6.1.2.1.2.2.1.7",    # - Administrative Status
            "1.3.6.1.2.1.2.2.1.8",    # - Operating Ststus
            "1.3.6.1.2.1.2.2.1.10",   # - In Bytes
            "1.3.6.1.2.1.2.2.1.11",   # - In Unicast Packets
            "1.3.6.1.2.1.2.2.1.13",   # - In Discards
            "1.3.6.1.2.1.2.2.1.14",   # - In Errors
            "1.3.6.1.2.1.2.2.1.16",   # - Out Bytes
            "1.3.6.1.2.1.2.2.1.17",   # - Out Unicast Packets
            "1.3.6.1.2.1.2.2.1.19",   # - Out Discards
            "1.3.6.1.2.1.2.2.1.20",   # - Out Errors
            "1.3.6.1.2.1.2.2.1.21",   # - Out Queue Length
            "1.3.6.1.2.1.31.1.1.1.2", # - In Multicast Packets
            "1.3.6.1.2.1.31.1.1.1.3", # - In Broadcast Packets
            "1.3.6.1.2.1.31.1.1.1.4", # - Out Multicast Packets
            "1.3.6.1.2.1.31.1.1.1.5"  # - Out Broadcast Packets
          ]}]
        hosts => [
          { host => "udp:172.16.1.4/161" community => "public" timeout => 10000 }
        ]
        oid_root_skip => 9
        interval => 60
        tags => [
          "snmp"
        ]
      }
    }
    filter {
      if [networkinterface] {
        split {
          field => "networkinterface"
        }
      }
    }
    output {
      if "snmp" in [tags] {
        elasticsearch {
          hosts => ["https://server:9200"]
          user => "user"
          password => "concept_testing"
          ssl_certificate_verification => false
          ilm_enabled => "true"
          ilm_rollover_alias => "snmp"
          ilm_pattern => "{now/d}-000001"
        }
      }
    }

Error from container:

[ERROR] 2021-10-27 13:06:23.207 [[main]<snmp] snmp - error invoking table operation on OID: networkinterface, ignoring {:exception=>#<LogStash::SnmpClientError: error sending snmp table request to target 172.16.1.4/161: Request timed out.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.8/lib/logstash/inputs/snmp/base_client.rb:105:in `block in table'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.8/lib/logstash/inputs/snmp/base_client.rb:100:in `table'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.8/lib/logstash/inputs/snmp.rb:189:in `block in run'", "org/jruby/RubyArray.java:1820:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.8/lib/logstash/inputs/snmp.rb:187:in `block in run'", "org/jruby/RubyArray.java:1820:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.8/lib/logstash/inputs/snmp.rb:167:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.8/lib/logstash/inputs/snmp.rb:327:in `every'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-snmp-1.2.8/lib/logstash/inputs/snmp.rb:166:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'"]}
2

Are you able to run a snmpwalk directly from the container to see if it is not a connection issue?

Try to ssh into the container and run snmpwalk to the same host and see if you get any response, if this is not work outside logstash, then it could really be a connection issue.

3

Unfortunately, snmpwalk doesn't exist on that container (docker.elastic.co/logstash/logstash-oss:7.15.1). I don't really know how to diagnose that other than by trying the same version of logstash on a VM (but that all works fine, which is the most confusing part).

The SNMP walk does work though, it's only the tables piece of the configuration that doesn't work with that odd timeout error. I can confirm that the container can see the machine i'm using to test with (as the snmp walk part of the config works), as well as ping and other things working from within the container to the test SNMP endpoint.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.