Custom Logstash pipeline configuration file

Elastic Stack Logstash
1

Hi,

I want to make a custom pipeline using SNMP walk plugin.
I need some help in the first steps to properly understand syntax.

So far I have something like this

filter {
        mutate {
                add_field => {"host.hostname" => "NameOfDevice"}
                rename => {"host" => "host.ip"}
        }

}

I am adding the field of a device to the pipeline(because I don't want to query It every time and the name should not change) and I'm changing the "host" field to "host.ip"

Let's say this is what I get from SNMP walk input.

{
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.6" => 100000000,
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.4" => 1000000000,
}

Have can I instruct logstash to pare this log in such a way It would create the below document.

{ 
  {
    "cisco.device.interface.number" : "6",
    "cisco.device.interface.speed" : 100000000
  },
  {
    "cisco.device.interface.number" : "4",
    "cisco.device.interface.speed" : 1000000000
  }
}

The expected dashboard output from this document should be for example any table that shows speed for every interface by given "host.ip"

2

I would do that in ruby

    ruby {
        code => '
            a = []
            event.to_hash.each { |k, v|
                if k =~ /^iso\.org\.dod\.internet\.mgmt\.mib-2\.interfaces\.ifTable\.ifEntry\.ifSpeed\.\d+$/
                    matches = k.scan(/(\d+)$/)
                    a << { "cisco.device.interface.number" => matches[0][0], "cisco.device.interface.speed" => v }
                end
            }
            if a != []
                event.set("someField", a)
            end
        '
    }
3

I tried your code and this is the output I got back(I have removed the part on interface number because I will later add interface name so I don't need the number)

{
                                                           "host.hostname" => "Hostname.domain",
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.1" => 0,
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.3" => 1000000000,
                                                                 "host.ip" => "10.10.10.10",
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.4" => 1000000000,
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.6" => 100000000,
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.7" => 100000000,
                                                              "@timestamp" => 2020-10-15T23:32:56.364Z,
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.5" => 100000000,
                                                                    "type" => "snmp",
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.2" => 0,
    "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.8" => 0,
                                                               "someField" => [
        [0] {
            "cisco.device.interface.speed" => 0
        },
        [1] {
            "cisco.device.interface.speed" => 1000000000
        },
        [2] {
            "cisco.device.interface.speed" => 1000000000
        },
        [3] {
            "cisco.device.interface.speed" => 100000000
        },
        [4] {
            "cisco.device.interface.speed" => 100000000
        },
        [5] {
            "cisco.device.interface.speed" => 100000000
        },
        [6] {
            "cisco.device.interface.speed" => 0
        },
        [7] {
            "cisco.device.interface.speed" => 0
        }
    ]
}

I still see the "iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifSpeed.1" etc. fields I tried to remove them with

mutate {
                remove_field => [ "%{.ifSpeed}"]
        }

after the ruby filter sequence but It does not seem to work.

I can see that your output is part of "someField" array.
Should in this case "someField" be cisco.device.interface and the speed itself just speed?

4

mutate+remove_field does not process wildcards or regexps. You could use prune+blacklist_names, or add

event.remove(k)

to the loop.

I have no insight into what final format you want the data in. I am just trying to suggest ideas of how to get there.

5

event.remove(k) did the work thanks!
I have tried to edit your code to achieve what I think is what I need.

I have made the below change

  code => '
        a = []
        event.to_hash.each { |k, v|
            if k =~ /^.*?ifSpeed\.\d+$/
                matches = k.scan(/(\d+)$/)
                a << { "speed" => v }
                event.remove(k)
             end
            if k =~ /^.*?ifName\.\d+$/
                matches = k.scan(/(\d+)$/)
                a << { "name" => v }
                event.remove(k)
             end

            }
        if a != []
            event.set("cisco.device.interface", a)
        end
    '

 [...]
  },
    [14] {
        "speed" => 0
    },
    [15] {
        "name" => "Virtual254"
    }
],

But the output is not what I expect.

How can I make It look like this?

[ 1] {
"name" => "Ethernet0/3",
"speed" => 100000000
},

6

Thank you so much for your efforts.
I get a lot from this.

I got my answer in this GitHub thread.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.