Great horny toads I SOLVED IT!!!!!
I was reading this post: Logstash Pipeline from 6.1 not working 6.2.1 - #40 by wwalker
...because I really, really needed to fix this so I started poking at random (but related) things in a desperate attempt to figure it out. (I don't know about you, but my brain has this bad habit of not letting me sleep properly when it's trying to solve a problem...)
Anyway, this comment from the above thread caught my attention:
I got the same bug, but my logstash 6.2.2 is installed on CentOS7.
And the same, if I sharp the queue.type: persisted in the pipelines.yml, it works.
I distinctly remember this working back in January back when I was using ELK 6.1, so on a lark I tried commenting out queue.type: persisted in my logstash.yml file and suddenly the garbled SNMP problem went away.
This is the very first event I captured after disabling persistent queues...all fields look correct now:
"SNMPv2-MIB::sysName.0" => "pro-pts",
"RFC1065-SMI::enterprises.11610.435.11281.1.11.1.3.11397.6467" => "ptsd",
"RFC1065-SMI::enterprises.11610.435.5213.1.2.1.0" => "ProActivePTS",
"RFC1065-SMI::enterprises.11610.435.11281.1.11.1.12.11397.6467" => "43",
"customer_name" => "poc_2",
"@version" => "1",
"@timestamp" => 2018-04-17T20:47:12.629Z,
"RFC1065-SMI::enterprises.11610.435.11281.1.11.1.5.11397.6467" => "1",
"type" => "snmptrap",
"SNMPv2-MIB::snmpTrapOID.0" => "RFC1065-SMI::enterprises.11610.6799.3.4.0.5",
"DISMAN-EVENT-MIB::sysUpTimeInstance" => "27 days, 01:51:09.00",
"RFC1065-SMI::enterprises.11610.6799.1.10.0" => "4",
"RFC1065-SMI::enterprises.11610.435.11281.1.11.1.6.11397.6467" => "1",
"hostname" => "172.16.1.13"
FAN-FUGU-TASTIC!
Now...what's all this about persistent queues messing things up? I kinda need those as much as I need to be able to ingest SNMP.
Hot damn I'm going to actually sleep tonight!