# How to remove document I logstash pipilnie that has no usefull fields **URL:** \ **Category:** Logstash\ **Created:** [October 25, 2020, 5:58pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243 "2020-10-25T17:58:05Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Post date:** [October 25, 2020, 5:58pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/1 "2020-10-25T17:58:05Z")
Hello, I use the logstash SNMP plugin to query my device. After that, I do some split and field renaming as below. ``` input { snmp { interval => 5 hosts => [ {host => "udp:ip/port" community => "secret"} ] tables => [ { name => "interfaces" columns => [ "1.3.6.1.2.1.2.1", "1.3.6.1.2.1.2.2.1.1", "1.3.6.1.2.1.2.2.1.3", "1.3.6.1.2.1.2.2.1.4", "1.3.6.1.2.1.2.2.1.5", "1.3.6.1.2.1.2.2.1.6", "1.3.6.1.2.1.2.2.1.7", "1.3.6.1.2.1.2.2.1.8", "1.3.6.1.2.1.2.2.1.9", "1.3.6.1.2.1.2.2.1.10", "1.3.6.1.2.1.2.2.1.11", "1.3.6.1.2.1.2.2.1.12", "1.3.6.1.2.1.2.2.1.13", "1.3.6.1.2.1.2.2.1.14", "1.3.6.1.2.1.2.2.1.15", "1.3.6.1.2.1.2.2.1.16", "1.3.6.1.2.1.2.2.1.17", "1.3.6.1.2.1.2.2.1.18", "1.3.6.1.2.1.2.2.1.19", "1.3.6.1.2.1.2.2.1.20", "1.3.6.1.2.1.31.1.1.1.1", "1.3.6.1.2.1.2.1" ] } ] add_field => { "host" => "%{[@metadata][host_address]}"} tags => ["snmp", "interface"] } } filter{ split { field => "interfaces" } mutate { remove_field => ["[interfaces]" ] # CPU # INTERFACE rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOutOctets]" => "cisco.device.system.interface.ifOutOctets"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifType]" => "cisco.device.system.interface.ifType"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed]" => "cisco.device.system.interface.ifSpeed"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.ifMIB.ifMIBObjects.ifXTable.ifXEntry.ifName]" => "cisco.device.system.interface.ifName"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOutUcastPkts]" => "cisco.device.system.interface.ifOutUcastPkts"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInNUcastPkts]" => "cisco.device.system.interface.ifInNUcastPkts"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInUcastPkts]" => "cisco.device.system.interface.ifInUcastPkts"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOutNUcastPkts]" => "cisco.device.system.interface.ifOutNUcastPkts"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInDiscards]" => "cisco.device.system.interface.ifInDiscards"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOutErrors]" => "cisco.device.system.interface.ifOutErrors"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOperStatus]" => "cisco.device.system.interface.ifOperStatus"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifIndex]" => "cisco.device.system.interface.ifIndex"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInErrors]" => "cisco.device.system.interface.ifInErrors"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOutDiscards]" => "cisco.device.system.interface.ifOutDiscards"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets]" => "cisco.device.system.interface.ifInOctets"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifPhysAddress]" => "cisco.device.system.interface.ifPhysAddress"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifLastChange]" => "cisco.device.system.interface.ifLastChange"} rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifMtu]" => "cisco.device.system.interface.ifMtu"} # HOST rename => { "host" => "host.ip"} add_field => { "host.hostname" => "Hostname" } } # Adding status explanation if 1 == [cisco.device.system.interface.ifOperStatus] { mutate { add_field => { "cisco.device.system.interface.ifOperStatusText" => "up" }} } else if 2 == [cisco.device.system.interface.ifOperStatus] { mutate { add_field => { "cisco.device.system.interface.ifOperStatusText" => "down" }} } else if 3 == [cisco.device.system.interface.ifOperStatus] { mutate { add_field => { "cisco.device.system.interface.ifOperStatusText" => "testing" }} } else if 4 == [cisco.device.system.interface.ifOperStatus] { mutate { add_field => { "cisco.device.system.interface.ifOperStatusText" => "unknown" }} } else if 5 == [cisco.device.system.interface.ifOperStatus] { mutate { add_field => { "cisco.device.system.interface.ifOperStatusText" => "notPresent" }} } else if 6 == [cisco.device.system.interface.ifOperStatus] { mutate { add_field => { "cisco.device.system.interface.ifOperStatusText" => "lowerLayerDown" }} } } output { # file { # path => "/etc/logstash/conf.d/tests/snmp.txt" # } stdout { codec => rubydebug } } ``` The output looks like below. ``` [...] { "@version" => "1", "cisco.device.system.interface.ifOutOctets" => 0, "cisco.device.system.interface.ifOperStatusText" => "down", "host.ip" => "ip", "cisco.device.system.interface.ifOutUcastPkts" => 0, "cisco.device.system.interface.ifInOctets" => 0, "cisco.device.system.interface.ifInErrors" => 0, "cisco.device.system.interface.ifInDiscards" => 0, "cisco.device.system.interface.ifOutDiscards" => 0, "cisco.device.system.interface.ifInNUcastPkts" => 0, "tags" => [ [0] "snmp", [1] "interface" ], "cisco.device.system.interface.ifOperStatus" => 2, "host.hostname" => "Hostname", "cisco.device.system.interface.ifMtu" => 1500, "cisco.device.system.interface.ifPhysAddress" => "00:00:00:00:00:00", "cisco.device.system.interface.ifType" => 6, "cisco.device.system.interface.ifName" => "management", "cisco.device.system.interface.ifSpeed" => 100000000, "cisco.device.system.interface.ifInUcastPkts" => 128, "cisco.device.system.interface.ifOutNUcastPkts" => 0, "cisco.device.system.interface.ifLastChange" => 400, "@timestamp" => 2020-10-25T17:54:17.809Z, "cisco.device.system.interface.ifOutErrors" => 0, "cisco.device.system.interface.ifIndex" => 7 } { "host.hostname" => "Hostname", "@timestamp" => 2020-10-25T17:54:17.809Z, "host.ip" => "Ip", "tags" => [ [0] "snmp", [1] "interface" ], "@version" => "1" } ``` I want to instruct logstash to delete this last document hance it does not contains any useful information. How can I do It and where does it come from? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 25, 2020, 7:24pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/2 "2020-10-25T19:24:33Z")
I would guess there is an empty entry in the interfaces array that the snmp input emits. I would remove that using ruby before doing the split. ---
**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Post date:** [October 26, 2020, 10:14pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/3 "2020-10-26T22:14:36Z")
So I notice that an empty document is a leftover of a document that at the beginning had an interfaces field. I removed the after a split because they are no longer necessary. How would you delete this document using ruby? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 26, 2020, 10:28pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/4 "2020-10-26T22:28:54Z")
In ruby you would call event.cancel to delete an entire event. There is an example [here](https://discuss.elastic.co/t/logstash-exclude-separator-lines-in-logs/175071/2). ---
**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Post date:** [October 26, 2020, 10:44pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/5 "2020-10-26T22:44:25Z")
``` ruby { code => ' if event.get("interfaces") event.cancel end ' } ``` This is deleting the whole event. Can I somehow narrow this just to a document containing that field? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 26, 2020, 10:54pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/6 "2020-10-26T22:54:58Z")
Yes, you can write arbitrary code to evaluate the fields of the event to see whether or not you want to retain it. ---
**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Post date:** [October 26, 2020, 10:57pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/7 "2020-10-26T22:57:24Z")
Can you pass me some resource to learn from? I have never coded in ruby nor in ruby in logstash. ---
**Author:** ![Alex\_Marquardt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_marquardt/32/42925_2.png) [@Alex\_Marquardt](https://discuss.elastic.co/u/Alex_Marquardt)\ **Post date:** [October 26, 2020, 11:33pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/8 "2020-10-26T23:33:18Z")
Perhaps you could use a conditional and the drop filter to drop the event. [https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html) ---
**Author:** ![Adriann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriann/32/77780_2.png) [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Post date:** [October 26, 2020, 11:53pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/9 "2020-10-26T23:53:44Z")
> [@Adriann](#): > > ` "1.3.6.1.2.1.2.1",` In my configuration file, I had this above oid and the additional document was related to this. It was some leftover form testing solution show basically the deleting it and the split function was my solution. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 23, 2020, 11:53pm UTC](https://discuss.elastic.co/t/how-to-remove-document-i-logstash-pipilnie-that-has-no-usefull-fields/253243/10 "2020-11-23T23:53:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.