Logstash SNMP input plugin not seeing metadata

Elastic Stack Logstash
1

I am trying to implement the SNMP input plugin to gther network data however when I try and access the metadata it returns the line of code not the data. I even tried copying the example from the documentation and get the same result
Here is the code I am using

input {
  snmp {
    tables => [{
                "name" => "ifTable"
                "columns" => ["1.3.6.1.2.1.2.2.1.1","1.3.6.1.2.1.2.2.1.2","1.3.6.1.2.1.2.2.1.3","1.3.6.1.2.1.2.2.1.4","1.3.6.1.2.1.2.2.1.5","1.3.6.1.2.1.2.2.1.6","1.3.6.1.2.1.2.2.1.7","1.3.6.1.2.1.2.2.1.8","1.3.6.1.2.1.2.2.1.9","1.3.6.1.2.1.2.2.1.10","1.3.6.1.2.1.2.2.1.11","1.3.6.1.2.1.2.2.1.12","1.3.6.1.2.1.2.2.1.13","1.3.6.1.2.1.2.2.1.14","1.3.6.1.2.1.2.2.1.15","1.3.6.1.2.1.2.2.1.16","1.3.6.1.2.1.2.2.1.17","1.3.6.1.2.1.2.2.1.18","1.3.6.1.2.1.2.2.1.19","1.3.6.1.2.1.2.2.1.20","1.3.6.1.2.1.2.2.1.21","1.3.6.1.2.1.2.2.1.22"]
                }]
    hosts => [{
                host => "udp:10.30.1.1/161" community => "public"  version => "2c"  retries => 2  timeout => 1000
              }]
    interval => 15
    oid_root_skip => 9
    add_field => {host => "%{[@metadata][host_protocol]}:%{[@metadata][host_address]}/%{[@metadata][host_port]},%{[@metadata][host_community]}"}
  }
}

output {

 stdout {
	codec => rubydebug
  }
}

and I get all the expected data returned except the metadata as seen in the response below. I have only copied the last column down as everything above it is as expected

        [15] {
                 "ifInDiscards" => 0,
               "ifOutUcastPkts" => 575106999,
               "ifInNUcastPkts" => 72251508,
                "ifOutDiscards" => 0,
                  "ifOutOctets" => 2870510397,
                "ifPhysAddress" => "d0:21:f9:44:0a:f8",
                      "ifSpeed" => 1000000000,
                  "ifOutErrors" => 0,
                "ifAdminStatus" => 1,
                    "ifOutQLen" => 0,
                   "ifSpecific" => "0.0",
                 "ifLastChange" => 250331528,
                        "index" => "16",
                   "ifInErrors" => 1,
                "ifInUcastPkts" => 655549316,
                       "ifType" => 6,
            "ifInUnknownProtos" => 0,
                        "ifMtu" => 1518,
                   "ifInOctets" => 1118489128,
                 "ifOperStatus" => 1,
              "ifOutNUcastPkts" => 31099656,
                      "ifIndex" => 16,
                      "ifDescr" => "Port 16"
        }
    ],
          "host" => "%{[@metadata][host_protocol]}:%{[@metadata][host_address]}/%{[@metadata][host_port]},%{[@metadata][host_community]}",
    "@timestamp" => 2023-12-18T01:57:58.377288900Z

It aso seems to add a comma to the end of the string returned

2

Perhaps you have ECS enabled. Does

add_field => {host => "%{[@metadata][input][snmp][host_protocol]}:%{[@metadata][input][snmp][host_address]}/%{[@metadata][input][snmp][host_port]},%{[@metadata][input][snmp][host_community]}"}

work? (And yes, the 3-column table in the documentation seems to be formatted as a 4-column table, so nothing lines up. Not helpful.)

3

Thank you. You were right I had to disable ECS. I thought reading the documentation it only was applicable when using SNMP 3. I added the line
ecs_compatibility => disabled
and it worked as expected. Once again thank you.

4

I managed to get all the rest of the code to appear to work however when I came to add the second ruby filter which I want to use to calculate the actual data transfer it crashes and I cannot see why. I even tried it with just the first line and not the calculations but it crashes. I am not sure if I should have added this question here or created a new ticket. I am using the latest logstash on a windows server. My logstash conf file in its entirity is as follows

input {
  snmp {
    tables => [{
                "name" => "ifTable"
                "columns" => ["1.3.6.1.2.1.2.2.1.1","1.3.6.1.2.1.2.2.1.2","1.3.6.1.2.1.2.2.1.3","1.3.6.1.2.1.2.2.1.4","1.3.6.1.2.1.2.2.1.5","1.3.6.1.2.1.2.2.1.6","1.3.6.1.2.1.2.2.1.7","1.3.6.1.2.1.2.2.1.8","1.3.6.1.2.1.2.2.1.9","1.3.6.1.2.1.2.2.1.10","1.3.6.1.2.1.2.2.1.11","1.3.6.1.2.1.2.2.1.12","1.3.6.1.2.1.2.2.1.13","1.3.6.1.2.1.2.2.1.14","1.3.6.1.2.1.2.2.1.15","1.3.6.1.2.1.2.2.1.16","1.3.6.1.2.1.2.2.1.17","1.3.6.1.2.1.2.2.1.18","1.3.6.1.2.1.2.2.1.19","1.3.6.1.2.1.2.2.1.20","1.3.6.1.2.1.2.2.1.21","1.3.6.1.2.1.2.2.1.22"]
                }]
    hosts => [{
                host => "udp:10.30.1.1/161" community => "public"  version => "2c"  retries => 2  timeout => 1000
              }]
    interval => 15
    oid_root_skip => 9
    add_field => { "device_ip" => "%{[@metadata][host_address]}" "device_name" => "Unifi Security Gateway USG" "location" => "29.113131, -80.999226"} 
  }

  snmp {
    tables => [{
                "name" => "ifTable"
                "columns" => ["1.3.6.1.2.1.2.2.1.1","1.3.6.1.2.1.2.2.1.2","1.3.6.1.2.1.2.2.1.3","1.3.6.1.2.1.2.2.1.4","1.3.6.1.2.1.2.2.1.5","1.3.6.1.2.1.2.2.1.6","1.3.6.1.2.1.2.2.1.7","1.3.6.1.2.1.2.2.1.8","1.3.6.1.2.1.2.2.1.9","1.3.6.1.2.1.2.2.1.10","1.3.6.1.2.1.2.2.1.11","1.3.6.1.2.1.2.2.1.12","1.3.6.1.2.1.2.2.1.13","1.3.6.1.2.1.2.2.1.14","1.3.6.1.2.1.2.2.1.15","1.3.6.1.2.1.2.2.1.16","1.3.6.1.2.1.2.2.1.17","1.3.6.1.2.1.2.2.1.18","1.3.6.1.2.1.2.2.1.19","1.3.6.1.2.1.2.2.1.20","1.3.6.1.2.1.2.2.1.21","1.3.6.1.2.1.2.2.1.22"]
                }]
    hosts => [{
                host => "udp:10.30.1.20/161" community => "public"  version => "2c"  retries => 2  timeout => 1000
              }]
    interval => 15
    oid_root_skip => 9
    add_field => { "device_ip" => "%{[@metadata][host_address]}" "device_name" => "Unifi USW-Lite-16-POE (45W)" "location" => "29.113131, -80.999226"} 
  }
}

#Start of Filter

filter {

# Split "ifTable" array fields in events into individual events

  split {
    field => "ifTable"
  }

# Move nested key value to root level of the ifTable object
  
  ruby {
    code => '
      event.get("[ifTable]").each do |k, v|
        event.set(k, v)
      event.remove("[ifTable]")
      end
    '
  }

  mutate {
    convert => {
      "ifInOctets" => "integer"
      "ifOutOctets" => "integer"
      "ifSpeed" => "integer"
    }
  }

# Query the latest counters of the interface from elasticsearch
  
  elasticsearch {
    hosts => ["http://localhost:9200"]
    user => "elastic"
    password => "49oo2OcGWFp6E=BoLmd7"
    index => "network-usg-*"
    query_template => "./config/query_ifIndex_template.json"
    fields => { "@timestamp" => "previousDataPollTimestamp" "ifInOctets" => "previousDataPollInOctets" "ifOutOctets" => "previousDataPollOutOctets"}
  }

#Test value of lastpolicy variables

  if ([previousDataPollTimestamp] =~ /.+/ ){ 
    date {
      match => [ "previousDataPollTimestamp", "ISO8601" ]
      target => "previousDataPollTimestamp"
    }
  } else {
      mutate {
        copy => { "@timestamp" => "previousDataPollTimestamp" }
        copy => { "ifInOctets" => "previousDataPollInOctets" }
        copy => { "ifOutOctets" => "previousDataPollOutOctets" }
    }
  }

  mutate {
    convert => {
      "previousDataPollInOctets" => "integer"
      "previousDataPollOutOctets" => "integer"
    }
  }

**#Compute Bandwisth utilization  **
**  **
**  ruby {**
**    init => "require 'time'"**
**      code ​=> "**
**        event.set('timeDifference', Time.parse(event.get('@timestamp').to_s) - Time.parse(event.get('previousDataPollTimestamp').to_s))**
**#      if (!event.get('timeDifference').nil? && event.get('timeDifference') > 0 && !event.get('ifSpeed').nil? && event.get('ifSpeed') > 0)**
**#        event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')) / (event.get('timeDifference') * event.get('ifSpeed')));**
**#        event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets'))/ (event.get('timeDifference') * event.get('ifSpeed')));**
**#      elsif (!event.get('ifSpeed').nil? && event.get('ifSpeed') > 0)**
**#        event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')) / event.get('ifSpeed'));**
**#        event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets'))/ event.get('ifSpeed'));**
**#      else**
**#        event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')));**
**#        event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets')));**
**        end**
**      "**
**    }  **

  mutate {
    remove_field => [ "@version" ]
   }

#End of Filter

}

output {

  stdout {
	codec => rubydebug
  }

  elasticsearch {
	hosts => ["http://localhost:9200"]
	index => "network-usg-%{+YYY.MM.dd}"
	user => "elastic"
	password => "49oo2OcGWFp6E=BoLmd7"
  }

the error i get is as follows

[2023-12-21T15:30:02,455][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 97, column 12 (byte 3567) after filter {\r\n\r\n# Split \"ifTable\" array fields in events into individual events\r\n\r\n  split {\r\n    field => \"ifTable\"\r\n  }\r\n\r\n# Move nested key value to root level of the ifTable object\r\n  \r\n  ruby {\r\n    code => '\r\n      event.get(\"[ifTable]\").each do |k, v|\r\n        event.set(k, v)\r\n      event.remove(\"[ifTable]\")\r\n      end\r\n    '\r\n  }\r\n\r\n  mutate {\r\n    convert => {\r\n      \"ifInOctets\" => \"integer\"\r\n      \"ifOutOctets\" => \"integer\"\r\n      \"ifSpeed\" => \"integer\"\r\n    }\r\n  }\r\n\r\n# Query the latest counters of the interface from elasticsearch\r\n  \r\n  elasticsearch {\r\n    hosts => [\"http://localhost:9200\"]\r\n    user => \"elastic\"\r\n    password => \"49oo2OcGWFp6E=BoLmd7\"\r\n    index => \"network-usg-*\"\r\n    query_template => \"./config/query_ifIndex_template.json\"\r\n    fields => { \"@timestamp\" => \"previousDataPollTimestamp\" \"ifInOctets\" => \"previousDataPollInOctets\" \"ifOutOctets\" => \"previousDataPollOutOctets\"}\r\n  }\r\n\r\n#Test value of lastpolicy variables\r\n\r\n  if ([previousDataPollTimestamp] =~ /.+/ ){ \r\n    date {\r\n      match => [ \"previousDataPollTimestamp\", \"ISO8601\" ]\r\n      target => \"previousDataPollTimestamp\"\r\n    }\r\n  } else {\r\n      mutate {\r\n        copy => { \"@timestamp\" => \"previousDataPollTimestamp\" }\r\n        copy => { \"ifInOctets\" => \"previousDataPollInOctets\" }\r\n        copy => { \"ifOutOctets\" => \"previousDataPollOutOctets\" }\r\n    }\r\n  }\r\n\r\n  mutate {\r\n    convert => {\r\n      \"previousDataPollInOctets\" => \"integer\"\r\n      \"previousDataPollOutOctets\" => \"integer\"\r\n    }\r\n  }\r\n\r\n#Compute Bandwisth utilization  \r\n  \r\n  ruby {\r\n    init => \"require 'time'\"\r\n      code ", :backtrace=>["D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
[2023-12-21T15:30:02,484][INFO ][logstash.runner          ] Logstash shut down.
[2023-12-21T15:30:02,493][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:795) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:758) ~[jruby.jar:?]
        at D_3a_.Elastic_20_ELM_20_Network.logstash_minus_8_dot_11_dot_1.lib.bootstrap.environment.<main>(D:\Elastic ELM Network\logstash-8.11.1\lib\bootstrap\environment.rb:90) ~[?:?]

when I comment out the code => line in the second ruby filter, as shown below, it runs fine.

#Compute Bandwisth utilization  
  
  ruby {
    init => "require 'time'"
#      code ​=> "
#        event.set('timeDifference', Time.parse(event.get('@timestamp').to_s) - Time.parse(event.get('previousDataPollTimestamp').to_s))
#      if (!event.get('timeDifference').nil? && event.get('timeDifference') > 0 && !event.get('ifSpeed').nil? && event.get('ifSpeed') > 0)
#        event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')) / (event.get('timeDifference') * event.get('ifSpeed')));
#        event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets'))/ (event.get('timeDifference') * event.get('ifSpeed')));
#      elsif (!event.get('ifSpeed').nil? && event.get('ifSpeed') > 0)
#        event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')) / event.get('ifSpeed'));
#        event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets'))/ event.get('ifSpeed'));
#      else
#        event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')));
#        event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets')));
#        end
#      "
    }

and the result below is as expected minus the calculated values of course

{
                       "ifType" => 6,
               "ifInNUcastPkts" => 78039214,
                 "ifInDiscards" => 0,
               "ifOutUcastPkts" => 617356059,
                    "device_ip" => "10.30.1.20",
                  "ifOutOctets" => 821347340,
    "previousDataPollOutOctets" => 211956002,
                        "index" => "16",
                "ifPhysAddress" => "d0:21:f9:44:0a:f8",
                   "ifSpecific" => "0.0",
                   "ifInErrors" => 1,
                   "@timestamp" => 2023-12-21T20:53:32.237243900Z,
                    "ifOutQLen" => 0,
    "previousDataPollTimestamp" => 2023-12-21T16:56:37.083Z,
              "ifOutNUcastPkts" => 33193186,
                      "ifIndex" => 16,
                 "ifLastChange" => 250331528,
     "previousDataPollInOctets" => 132755338,
                     "location" => "29.113131, -80.999226",
                 "ifOperStatus" => 1,
                  "device_name" => "Unifi USW-Lite-16-POE (45W)",
            "ifInUnknownProtos" => 0,
                   "ifInOctets" => 384761415,
                        "ifMtu" => 1518,
                "ifInUcastPkts" => 711481747,
                      "ifDescr" => "Port 16",
                  "ifOutErrors" => 0,
                      "ifSpeed" => 1000000000,
                "ifOutDiscards" => 0,
                "ifAdminStatus" => 1
}
5

The configuration compiler is objecting to what comes immediately after "code ". I suggest you delete and retype the =>. Perhaps you copy and pasted a homoglyph for = from somewhere.

6

Thank you Badger I had spent hours trying to find out what the error was telling me. I ended up completely deleting the code and re-entering it and it worked. I still cannot quite work out how you determined it so if it is an easy explanation could you share it with me as I know you must be very busy seeing all the responses you have on the site.

7

The LogStash::ConfigurationError message includes the last part of the configuration that the compiler was able to parse. The last thing in that error message is code, so the thing that it is unable to parse is whatever appears in the configuration after code.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.