# Logstash SNMP input plugin not seeing metadata
**URL:** https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524
**Category:** Logstash
**Created:** [December 18, 2023, 2:00am UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524 "2023-12-18T02:00:57Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@bytelink](https://discuss.elastic.co/u/bytelink)
#### Post date: [December 18, 2023, 2:00am UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524/1 "2023-12-18T02:00:57Z")
I am trying to implement the SNMP input plugin to gther network data however when I try and access the metadata it returns the line of code not the data. I even tried copying the example from the documentation and get the same result
Here is the code I am using
```auto
input {
snmp {
tables => [{
"name" => "ifTable"
"columns" => ["1.3.6.1.2.1.2.2.1.1","1.3.6.1.2.1.2.2.1.2","1.3.6.1.2.1.2.2.1.3","1.3.6.1.2.1.2.2.1.4","1.3.6.1.2.1.2.2.1.5","1.3.6.1.2.1.2.2.1.6","1.3.6.1.2.1.2.2.1.7","1.3.6.1.2.1.2.2.1.8","1.3.6.1.2.1.2.2.1.9","1.3.6.1.2.1.2.2.1.10","1.3.6.1.2.1.2.2.1.11","1.3.6.1.2.1.2.2.1.12","1.3.6.1.2.1.2.2.1.13","1.3.6.1.2.1.2.2.1.14","1.3.6.1.2.1.2.2.1.15","1.3.6.1.2.1.2.2.1.16","1.3.6.1.2.1.2.2.1.17","1.3.6.1.2.1.2.2.1.18","1.3.6.1.2.1.2.2.1.19","1.3.6.1.2.1.2.2.1.20","1.3.6.1.2.1.2.2.1.21","1.3.6.1.2.1.2.2.1.22"]
}]
hosts => [{
host => "udp:10.30.1.1/161" community => "public" version => "2c" retries => 2 timeout => 1000
}]
interval => 15
oid_root_skip => 9
add_field => {host => "%{[@metadata][host_protocol]}:%{[@metadata][host_address]}/%{[@metadata][host_port]},%{[@metadata][host_community]}"}
}
}
output {
stdout {
codec => rubydebug
}
}
```
and I get all the expected data returned except the metadata as seen in the response below. I have only copied the last column down as everything above it is as expected
```auto
[15] {
"ifInDiscards" => 0,
"ifOutUcastPkts" => 575106999,
"ifInNUcastPkts" => 72251508,
"ifOutDiscards" => 0,
"ifOutOctets" => 2870510397,
"ifPhysAddress" => "d0:21:f9:44:0a:f8",
"ifSpeed" => 1000000000,
"ifOutErrors" => 0,
"ifAdminStatus" => 1,
"ifOutQLen" => 0,
"ifSpecific" => "0.0",
"ifLastChange" => 250331528,
"index" => "16",
"ifInErrors" => 1,
"ifInUcastPkts" => 655549316,
"ifType" => 6,
"ifInUnknownProtos" => 0,
"ifMtu" => 1518,
"ifInOctets" => 1118489128,
"ifOperStatus" => 1,
"ifOutNUcastPkts" => 31099656,
"ifIndex" => 16,
"ifDescr" => "Port 16"
}
],
"host" => "%{[@metadata][host_protocol]}:%{[@metadata][host_address]}/%{[@metadata][host_port]},%{[@metadata][host_community]}",
"@timestamp" => 2023-12-18T01:57:58.377288900Z
```
It aso seems to add a comma to the end of the string returned
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 18, 2023, 2:10am UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524/2 "2023-12-18T02:10:25Z")
Perhaps you have [ECS enabled](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-snmp.html#plugins-inputs-snmp-ecs). Does
```
add_field => {host => "%{[@metadata][input][snmp][host_protocol]}:%{[@metadata][input][snmp][host_address]}/%{[@metadata][input][snmp][host_port]},%{[@metadata][input][snmp][host_community]}"}
```
work? (And yes, the 3-column table in the documentation seems to be formatted as a 4-column table, so nothing lines up. Not helpful.)
---
### Author:  [@bytelink](https://discuss.elastic.co/u/bytelink)
#### Post date: [December 18, 2023, 4:58am UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524/3 "2023-12-18T04:58:43Z")
Thank you. You were right I had to disable ECS. I thought reading the documentation it only was applicable when using SNMP 3. I added the line
ecs\_compatibility =\> disabled
and it worked as expected. Once again thank you.
---
### Author:  [@bytelink](https://discuss.elastic.co/u/bytelink)
#### Post date: [December 21, 2023, 8:57pm UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524/4 "2023-12-21T20:57:09Z")
I managed to get all the rest of the code to appear to work however when I came to add the second ruby filter which I want to use to calculate the actual data transfer it crashes and I cannot see why. I even tried it with just the first line and not the calculations but it crashes. I am not sure if I should have added this question here or created a new ticket. I am using the latest logstash on a windows server. My logstash conf file in its entirity is as follows
```auto
input {
snmp {
tables => [{
"name" => "ifTable"
"columns" => ["1.3.6.1.2.1.2.2.1.1","1.3.6.1.2.1.2.2.1.2","1.3.6.1.2.1.2.2.1.3","1.3.6.1.2.1.2.2.1.4","1.3.6.1.2.1.2.2.1.5","1.3.6.1.2.1.2.2.1.6","1.3.6.1.2.1.2.2.1.7","1.3.6.1.2.1.2.2.1.8","1.3.6.1.2.1.2.2.1.9","1.3.6.1.2.1.2.2.1.10","1.3.6.1.2.1.2.2.1.11","1.3.6.1.2.1.2.2.1.12","1.3.6.1.2.1.2.2.1.13","1.3.6.1.2.1.2.2.1.14","1.3.6.1.2.1.2.2.1.15","1.3.6.1.2.1.2.2.1.16","1.3.6.1.2.1.2.2.1.17","1.3.6.1.2.1.2.2.1.18","1.3.6.1.2.1.2.2.1.19","1.3.6.1.2.1.2.2.1.20","1.3.6.1.2.1.2.2.1.21","1.3.6.1.2.1.2.2.1.22"]
}]
hosts => [{
host => "udp:10.30.1.1/161" community => "public" version => "2c" retries => 2 timeout => 1000
}]
interval => 15
oid_root_skip => 9
add_field => { "device_ip" => "%{[@metadata][host_address]}" "device_name" => "Unifi Security Gateway USG" "location" => "29.113131, -80.999226"}
}
snmp {
tables => [{
"name" => "ifTable"
"columns" => ["1.3.6.1.2.1.2.2.1.1","1.3.6.1.2.1.2.2.1.2","1.3.6.1.2.1.2.2.1.3","1.3.6.1.2.1.2.2.1.4","1.3.6.1.2.1.2.2.1.5","1.3.6.1.2.1.2.2.1.6","1.3.6.1.2.1.2.2.1.7","1.3.6.1.2.1.2.2.1.8","1.3.6.1.2.1.2.2.1.9","1.3.6.1.2.1.2.2.1.10","1.3.6.1.2.1.2.2.1.11","1.3.6.1.2.1.2.2.1.12","1.3.6.1.2.1.2.2.1.13","1.3.6.1.2.1.2.2.1.14","1.3.6.1.2.1.2.2.1.15","1.3.6.1.2.1.2.2.1.16","1.3.6.1.2.1.2.2.1.17","1.3.6.1.2.1.2.2.1.18","1.3.6.1.2.1.2.2.1.19","1.3.6.1.2.1.2.2.1.20","1.3.6.1.2.1.2.2.1.21","1.3.6.1.2.1.2.2.1.22"]
}]
hosts => [{
host => "udp:10.30.1.20/161" community => "public" version => "2c" retries => 2 timeout => 1000
}]
interval => 15
oid_root_skip => 9
add_field => { "device_ip" => "%{[@metadata][host_address]}" "device_name" => "Unifi USW-Lite-16-POE (45W)" "location" => "29.113131, -80.999226"}
}
}
#Start of Filter
filter {
# Split "ifTable" array fields in events into individual events
split {
field => "ifTable"
}
# Move nested key value to root level of the ifTable object
ruby {
code => '
event.get("[ifTable]").each do |k, v|
event.set(k, v)
event.remove("[ifTable]")
end
'
}
mutate {
convert => {
"ifInOctets" => "integer"
"ifOutOctets" => "integer"
"ifSpeed" => "integer"
}
}
# Query the latest counters of the interface from elasticsearch
elasticsearch {
hosts => ["http://localhost:9200"]
user => "elastic"
password => "49oo2OcGWFp6E=BoLmd7"
index => "network-usg-*"
query_template => "./config/query_ifIndex_template.json"
fields => { "@timestamp" => "previousDataPollTimestamp" "ifInOctets" => "previousDataPollInOctets" "ifOutOctets" => "previousDataPollOutOctets"}
}
#Test value of lastpolicy variables
if ([previousDataPollTimestamp] =~ /.+/ ){
date {
match => ["previousDataPollTimestamp", "ISO8601"]
target => "previousDataPollTimestamp"
}
} else {
mutate {
copy => { "@timestamp" => "previousDataPollTimestamp" }
copy => { "ifInOctets" => "previousDataPollInOctets" }
copy => { "ifOutOctets" => "previousDataPollOutOctets" }
}
}
mutate {
convert => {
"previousDataPollInOctets" => "integer"
"previousDataPollOutOctets" => "integer"
}
}
**#Compute Bandwisth utilization**
****
**ruby {**
**init => "require 'time'"**
**code => "**
**event.set('timeDifference', Time.parse(event.get('@timestamp').to_s) - Time.parse(event.get('previousDataPollTimestamp').to_s))**
**# if (!event.get('timeDifference').nil? && event.get('timeDifference') > 0 && !event.get('ifSpeed').nil? && event.get('ifSpeed') > 0)**
**# event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')) / (event.get('timeDifference') * event.get('ifSpeed')));**
**# event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets'))/ (event.get('timeDifference') * event.get('ifSpeed')));**
**# elsif (!event.get('ifSpeed').nil? && event.get('ifSpeed') > 0)**
**# event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')) / event.get('ifSpeed'));**
**# event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets'))/ event.get('ifSpeed'));**
**# else**
**# event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')));**
**# event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets')));**
**end**
**"**
**}**
mutate {
remove_field => ["@version"]
}
#End of Filter
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["http://localhost:9200"]
index => "network-usg-%{+YYY.MM.dd}"
user => "elastic"
password => "49oo2OcGWFp6E=BoLmd7"
}
```
the error i get is as follows
```auto
[2023-12-21T15:30:02,455][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"=>\" at line 97, column 12 (byte 3567) after filter {\r\n\r\n# Split \"ifTable\" array fields in events into individual events\r\n\r\n split {\r\n field => \"ifTable\"\r\n }\r\n\r\n# Move nested key value to root level of the ifTable object\r\n \r\n ruby {\r\n code => '\r\n event.get(\"[ifTable]\").each do |k, v|\r\n event.set(k, v)\r\n event.remove(\"[ifTable]\")\r\n end\r\n '\r\n }\r\n\r\n mutate {\r\n convert => {\r\n \"ifInOctets\" => \"integer\"\r\n \"ifOutOctets\" => \"integer\"\r\n \"ifSpeed\" => \"integer\"\r\n }\r\n }\r\n\r\n# Query the latest counters of the interface from elasticsearch\r\n \r\n elasticsearch {\r\n hosts => [\"http://localhost:9200\"]\r\n user => \"elastic\"\r\n password => \"49oo2OcGWFp6E=BoLmd7\"\r\n index => \"network-usg-*\"\r\n query_template => \"./config/query_ifIndex_template.json\"\r\n fields => { \"@timestamp\" => \"previousDataPollTimestamp\" \"ifInOctets\" => \"previousDataPollInOctets\" \"ifOutOctets\" => \"previousDataPollOutOctets\"}\r\n }\r\n\r\n#Test value of lastpolicy variables\r\n\r\n if ([previousDataPollTimestamp] =~ /.+/ ){ \r\n date {\r\n match => [\"previousDataPollTimestamp\", \"ISO8601\"]\r\n target => \"previousDataPollTimestamp\"\r\n }\r\n } else {\r\n mutate {\r\n copy => { \"@timestamp\" => \"previousDataPollTimestamp\" }\r\n copy => { \"ifInOctets\" => \"previousDataPollInOctets\" }\r\n copy => { \"ifOutOctets\" => \"previousDataPollOutOctets\" }\r\n }\r\n }\r\n\r\n mutate {\r\n convert => {\r\n \"previousDataPollInOctets\" => \"integer\"\r\n \"previousDataPollOutOctets\" => \"integer\"\r\n }\r\n }\r\n\r\n#Compute Bandwisth utilization \r\n \r\n ruby {\r\n init => \"require 'time'\"\r\n code ", :backtrace=>["D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
[2023-12-21T15:30:02,484][INFO][logstash.runner] Logstash shut down.
[2023-12-21T15:30:02,493][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:795) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:758) ~[jruby.jar:?]
at D_3a_.Elastic_20_ELM_20_Network.logstash_minus_8_dot_11_dot_1.lib.bootstrap.environment.(D:\Elastic ELM Network\logstash-8.11.1\lib\bootstrap\environment.rb:90) ~[?:?]
```
when I comment out the code =\> line in the second ruby filter, as shown below, it runs fine.
```auto
#Compute Bandwisth utilization
ruby {
init => "require 'time'"
# code => "
# event.set('timeDifference', Time.parse(event.get('@timestamp').to_s) - Time.parse(event.get('previousDataPollTimestamp').to_s))
# if (!event.get('timeDifference').nil? && event.get('timeDifference') > 0 && !event.get('ifSpeed').nil? && event.get('ifSpeed') > 0)
# event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')) / (event.get('timeDifference') * event.get('ifSpeed')));
# event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets'))/ (event.get('timeDifference') * event.get('ifSpeed')));
# elsif (!event.get('ifSpeed').nil? && event.get('ifSpeed') > 0)
# event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')) / event.get('ifSpeed'));
# event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets'))/ event.get('ifSpeed'));
# else
# event.set('inboundUtilization', (event.get('ifInOctets') - event.get('previousDataPollInOctets')));
# event.set('outboundUtilization', (event.get('ifOutOctets') - event.get('previousDataPollOutOctets')));
# end
# "
}
```
and the result below is as expected minus the calculated values of course
```auto
{
"ifType" => 6,
"ifInNUcastPkts" => 78039214,
"ifInDiscards" => 0,
"ifOutUcastPkts" => 617356059,
"device_ip" => "10.30.1.20",
"ifOutOctets" => 821347340,
"previousDataPollOutOctets" => 211956002,
"index" => "16",
"ifPhysAddress" => "d0:21:f9:44:0a:f8",
"ifSpecific" => "0.0",
"ifInErrors" => 1,
"@timestamp" => 2023-12-21T20:53:32.237243900Z,
"ifOutQLen" => 0,
"previousDataPollTimestamp" => 2023-12-21T16:56:37.083Z,
"ifOutNUcastPkts" => 33193186,
"ifIndex" => 16,
"ifLastChange" => 250331528,
"previousDataPollInOctets" => 132755338,
"location" => "29.113131, -80.999226",
"ifOperStatus" => 1,
"device_name" => "Unifi USW-Lite-16-POE (45W)",
"ifInUnknownProtos" => 0,
"ifInOctets" => 384761415,
"ifMtu" => 1518,
"ifInUcastPkts" => 711481747,
"ifDescr" => "Port 16",
"ifOutErrors" => 0,
"ifSpeed" => 1000000000,
"ifOutDiscards" => 0,
"ifAdminStatus" => 1
}
```
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 21, 2023, 9:28pm UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524/5 "2023-12-21T21:28:35Z")
> [@bytelink](#):
>
> ```auto
> init => "require 'time'"
> # code => "
>
> ```
The configuration compiler is objecting to what comes immediately after "code ". I suggest you delete and retype the =\>. Perhaps you copy and pasted a homoglyph for = from somewhere.
---
### Author:  [@bytelink](https://discuss.elastic.co/u/bytelink)
#### Post date: [December 22, 2023, 4:44pm UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524/6 "2023-12-22T16:44:41Z")
Thank you Badger I had spent hours trying to find out what the error was telling me. I ended up completely deleting the code and re-entering it and it worked. I still cannot quite work out how you determined it so if it is an easy explanation could you share it with me as I know you must be very busy seeing all the responses you have on the site.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 22, 2023, 5:04pm UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524/7 "2023-12-22T17:04:26Z")
> [@bytelink](#):
>
> [2023-12-21T15:30:02,455][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [\t\r\n], "#", "=\>" at line 97, column 12 (byte 3567) after filter {\r\n\r\n# Split "ifTable" array fields in events into individual events\r\n\r\n split {\r\n field =\> "ifTable"\r\n }\r\n\r\n# Move nested key value to root level of the ifTable object\r\n \r\n ruby {\r\n code =\> '\r\n event.get("[ifTable]").each do |k, v|\r\n event.set(k, v)\r\n event.remove("[ifTable]")\r\n end\r\n '\r\n }\r\n\r\n mutate {\r\n convert =\> {\r\n "ifInOctets" =\> "integer"\r\n "ifOutOctets" =\> "integer"\r\n "ifSpeed" =\> "integer"\r\n }\r\n }\r\n\r\n# Query the latest counters of the interface from elasticsearch\r\n \r\n elasticsearch {\r\n hosts =\> ["[http://localhost:9200](http://localhost:9200)"]\r\n user =\> "elastic"\r\n password =\> "49oo2OcGWFp6E=BoLmd7"\r\n index =\> "network-usg-\*"\r\n query\_template =\> "./config/query\_ifIndex\_template.json"\r\n fields =\> { "@timestamp" =\> "previousDataPollTimestamp" "ifInOctets" =\> "previousDataPollInOctets" "ifOutOctets" =\> "previousDataPollOutOctets"}\r\n }\r\n\r\n#Test value of lastpolicy variables\r\n\r\n if ([previousDataPollTimestamp] =~ /.+/ ){ \r\n date {\r\n match =\> ["previousDataPollTimestamp", "ISO8601"]\r\n target =\> "previousDataPollTimestamp"\r\n }\r\n } else {\r\n mutate {\r\n copy =\> { "@timestamp" =\> "previousDataPollTimestamp" }\r\n copy =\> { "ifInOctets" =\> "previousDataPollInOctets" }\r\n copy =\> { "ifOutOctets" =\> "previousDataPollOutOctets" }\r\n }\r\n }\r\n\r\n mutate {\r\n convert =\> {\r\n "previousDataPollInOctets" =\> "integer"\r\n "previousDataPollOutOctets" =\> "integer"\r\n }\r\n }\r\n\r\n#Compute Bandwisth utilization \r\n \r\n **ruby {\r\n init =\> "require 'time'"\r\n code**", :backtrace=\>["D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "D:/Elastic ELM Network/logstash-8.11.1/logstash-core/lib/logstash/agent.rb:386:in `block in converge\_state'"]}
The LogStash::ConfigurationError message includes the last part of the configuration that the compiler was able to parse. The last thing in that error message is **code** , so the thing that it is unable to parse is whatever appears in the configuration after **code**.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 19, 2024, 5:04pm UTC](https://discuss.elastic.co/t/logstash-snmp-input-plugin-not-seeing-metadata/349524/8 "2024-01-19T17:04:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.