SNMPTrap not reading snmp traffic through logstash

Elastic Stack Logstash
1

I am trying to send SNMP traffic through Logstash using the SNMPtrap filter. I can see traffic coming into the machine that Logstash is on using TCPdump, and it is directed at the IP and port 162. I have run Logstash as root so it can listen on 162, and have enabled the IPtable rules so that SNMP traffic can come in. My input configuration looks like:

input {
snmptrap{
type => "snmptrap"
community => "myCommunity"
host => "*"
port => 162
yamlmibdir => "/application/nfmoss/vendors/logstash/vendor/bundle/jruby/1.9/gems/snmp-1.2.0/data/ruby/snmp/mibs"
}
http{
host => ""
port => 5044
}
}

Is there a configuration setting that I am missing? On Logstash's side, when I run it in verbose and debug mode, I don't see any visible errors, but no messages are outputted when an SNMP packet comes in. I am sure that Logstash is running and is working with Elasticsearch because I can still pass a successful curl request to the http input configuration. Is there maybe a setting that I'm missing that is filtering the packet between coming into the port and actually getting to Logstash?

2

The community string is correct?

3

Yes, it is the same name in the packets coming in as well as what we named it on the traffic-sender.

4

I am seeing traffic generated to ports snmptrap (162), lxi-eventsc (5044), and swxadmin (5043). I changed my config port to listen on all of these and still nothing in Logstash.

5

I am confused as to what you have configured.

This worked fine for me (I used the default 1062 port, rather than the privileged 162 - but should this should not matter)

Logstash config

input {
  snmptrap {}
}

filter {
}

output {
  stdout {
    codec => rubydebug { metadata => true }
  }
}

Test trap command

$ snmptrap -v 2c -c public localhost:1062 '' .1.3.6.1.6.3.1.1.5.3 ifIndex i 2 ifAdminStatus i 1 ifOperStatus i 1

Logstash stdout

logstash-2.3.2$ bin/logstash -f ../logstash-snmp.config
Settings: Default pipeline workers: 4
Pipeline main started
{
                      "message" => "#<SNMP::SNMPv2_Trap:0x37c613c @request_id=1803122945, @error_index=0, @error_status=0, @source_ip=\"127.0.0.1\", @varbind_list=[#<SNMP::VarBind:0x42416b07 @name=[1.3.6.1.2.1.1.3.0], @value=#<SNMP::TimeTicks:0x13dc79f2 @value=59480382>>, #<SNMP::VarBind:0x73c98f34 @name=[1.3.6.1.6.3.1.1.4.1.0], @value=[1.3.6.1.6.3.1.1.5.3]>, #<SNMP::VarBind:0x6bbec36b @name=[1.3.6.1.2.1.2.2.1.1], @value=#<SNMP::Integer:0x226e9448 @value=2>>, #<SNMP::VarBind:0x528d6a1e @name=[1.3.6.1.2.1.2.2.1.7], @value=#<SNMP::Integer:0x6d02ef54 @value=1>>, #<SNMP::VarBind:0x70bacc04 @name=[1.3.6.1.2.1.2.2.1.8], @value=#<SNMP::Integer:0x2620ca8f @value=1>>]>",
                         "host" => "127.0.0.1",
                     "@version" => "1",
                   "@timestamp" => "2016-07-25T16:03:32.834Z",
      "SNMPv2-MIB::sysUpTime.0" => "6 days, 21:13:23.82",
    "SNMPv2-MIB::snmpTrapOID.0" => "IF-MIB::linkDown",
              "IF-MIB::ifIndex" => "2",
        "IF-MIB::ifAdminStatus" => "1",
         "IF-MIB::ifOperStatus" => "1"
}

Maybe post the details of the trap from tcpdump?

6

I think the issue here is that somewhere between the interface and the application, the traffic is being dropped. I am on Centos7 and when I issue a similar snmptrap command it hangs before timing out.

A packet from tcpdump:
11:36:17.472584 IP *******.snmptrap > *******.snmptrap: C=myCommunity V2Trap(169) .1.3.6.1.2.1.1.3.0=232458448 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.6.3.1.1.5.4 .1.3.6.1.2.1.2.2.1.1.2107909=2107909 .1.3.6.1.4.1.6027.3.1.1.4.1.2="OSTATE_UP: Changed interface state to up: Fo 1/22/1" .1.3.6.1.4.1.6027.3.6.1.1.2.0=1219

7

iptables?

8

It's allowed in iptables, before putting in the rules TCPdump didnt get anything.

9

Seems like an environment issue. I just tried with Centos 7, logstash 2.3.4 - no issue. The only difference in my setup, I did not set the yamlmibdir.

input {
  snmptrap {
    type => "snmptrap"
    community => "myCommunity"
    port => 162
  }
}

filter {
}

output {
  stdout {
    codec => rubydebug { metadata => true }
  }
}

snmptrap -v 2c -c myCommunity 10.0.2.15:162 '' .1.3.6.1.6.3.1.1.5.4 .1.3.6.1.2.1.2.2.1.1.2107909 i 2107909 .1.3.6.1.4.1.6027.3.1.1.4.1.2 s "OSTATE_UP: Changed interface state to up: Fo 1/22/1" .1.3.6.1.4.1.6027.3.6.1.1.2.0 i 1219

sudo bin/logstash -f ../logstash-snmp.conf
Settings: Default pipeline workers: 1
Pipeline main started
{
                                     "message" => "#<SNMP::SNMPv2_Trap:0x46eb40c4 @request_id=219634030, @error_index=0, @error_status=0, @source_ip=\"10.0.2.15\", @varbind_list=[#<SNMP::VarBind:0x321338a6 @name=[1.3.6.1.2.1.1.3.0], @value=#<SNMP::TimeTicks:0x2aeea0cc @value=1672088>>, #<SNMP::VarBind:0x361f2835 @name=[1.3.6.1.6.3.1.1.4.1.0], @value=[1.3.6.1.6.3.1.1.5.4]>, #<SNMP::VarBind:0x821944d @name=[1.3.6.1.2.1.2.2.1.1.2107909], @value=#<SNMP::Integer:0x23dc5be7 @value=2107909>>, #<SNMP::VarBind:0x7c6b6790 @name=[1.3.6.1.4.1.6027.3.1.1.4.1.2], @value=\"OSTATE_UP: Changed interface state to up: Fo 1/22/1\">, #<SNMP::VarBind:0x33faa6a8 @name=[1.3.6.1.4.1.6027.3.6.1.1.2.0], @value=#<SNMP::Integer:0x37fddf66 @value=1219>>]>",
                                        "host" => "10.0.2.15",
                                    "@version" => "1",
                                  "@timestamp" => "2016-03-19T02:22:37.033Z",
                                        "type" => "snmptrap",
                     "SNMPv2-MIB::sysUpTime.0" => "04:38:40.88",
                   "SNMPv2-MIB::snmpTrapOID.0" => "IF-MIB::linkUp",
                     "IF-MIB::ifIndex.2107909" => "2107909",
    "SNMPv2-SMI::enterprises.6027.3.1.1.4.1.2" => "OSTATE_UP: Changed interface state to up: Fo 1/22/1",
    "SNMPv2-SMI::enterprises.6027.3.6.1.1.2.0" => "1219"
}
10

That's interesting. When I run that command it goes through to Logstash, but I get a Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2016.07.26", :_type=>"snmptrap", :_routing=>nil} with a lot more redundant info repeating the message. It doesn't go through to Elasticsearach. I am not sure why it is dropping it between Logstash and Elasticsearch, and why this packet goes through but the ones from the real switch do not. When I run tcpdump looking at the interface and filtering out only packets going to the SNMP ports, both types of packets come in looking the same.

tcp command:
tcpdump -n -i enp6s0f1 "udp and (dst port 161 or 162)"

I'm not sure what the second part of that command is though, the .1.3.6.1.6.3...., and maybe that is the difference?

11

I do not think it is being dropped, rather rejected from elasticsearch. If you are using a 2.x version of elasticsearch, you cannot have dots/periods in the field name. If you look at the very end of that long exception line from the logstash log file, it should read something like this:
"reason"=>"Field name [IF-MIB::ifIndex.2107909] cannot contain '.'"}}}, :level=>:warn}

(you can do something like this to get the last 150 characters of each line - grep -o ".\{150\}$" logstash.log)

This is someone problematic with how SNMP OID format (ex. SNMPv2-SMI::enterprises.6027.3.1.1.4.1.2 would need to be SNMPv2-SMI::enterprises_6027_3_1_1_4_1_2).

If this is a "dot used in the field name issue", there are two ways you could fix this.

  1. You can specify compatible field names for each oid that you expect to see in the MIB yaml files.
fieldname_1: .1.3.6.1.4.1.6027.3.6.1.1.2.0
fieldname_2: .1.3.6.1.4.1.6027.3.1.1.4.1.2


2. Use the logstash de_dot filter to remove dots from the field names.

12

Awesome, thank you so much. It turns out the error was Logstash was listening on IP address 10.x.x.x and when I tested, I was sending from 10.x.x.x. The switch was on 192.x.x.x, and for some reason Logstash was not taking in the data from the 192.x.x.x packets. I switched Logstash to listen on 192.x.x.x and now it works great. I ended up using the de_dot filter because our traps are going to be dynamic and won't always have the field names. Again, thanks for your help!