How to use dictionary with multidimensional array in Logstash?


#1

Hi,

I realize a project to collect NetFlow metric on Cisco router with Logstash.

With the NetFlow protocol, I collect host address, and only the OID of interfaces (like "1") and not names interfaces (like "Gi8" for example).

I would like to match host IP to convert the OID of interfaces to names interfaces, with use a dictionary file (interfaces.yaml) like this :

"HOST": "OID interface:Name interface"

"172.16.1.1": "1:Gi8"
"172.16.1.1": "2:Gi9"
"172.16.1.1": "3:Gi0"
"172.16.1.1": "4:Gi1"
"172.16.1.1": "5:Gi2"
"172.16.1.1": "6:Gi3"
"172.16.1.1": "7:Gi4"
"172.16.1.1": "8:Gi5"
"172.16.1.1": "9:Gi6"
"172.16.1.1": "10:Gi7"

And my logstash configuration is :

input {
    udp {
        port => 9995
        type => netflow
        codec => netflow {
            definitions => "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-master/lib/logstash/codecs/netflow/netflow.yaml"
            versions => [9]           
        }
    }
}
filter {
    translate {
        dictionary_path => "/etc/logstash/yaml/interfaces.yaml"  #It's the dictionary file
        field => "[host]" #[host] is the host address field
        destination => "array_input_interface"
        fallback => "unknown:unknown"
}
    mutate{
        split => [ "array_input_interface" , ":" ]
        add_field => [ "[netflow][input_interface]", "%{[array_input_interface][1]}" ] #[input_interface] is the field desired
        remove_field => [ "array_input_interface" ]
    }
}
output {
    stdout { codec => rubydebug }
    file {
        path => "/var/log/logstash/test" #I check metrics format with this file
    }
    elasticsearch {
        index => "logstash_netflow9-%{+YYYY.MM.dd}"
        hosts  => "127.0.0.1" #I export metrics in elasticsearch
    }
}

The problem is that only the last interface is always chosen : "Gi7" even if OID is different of 10. The match doesn't work and I don't know why..

Any idea ?

Thanks for help.

PS : I'm sorry for my language, but I have some difficulties with english language. I can to explain with more details if it's not clear


(Mark Walkom) #2

Does it work if you remove the quotes around the second value? As per https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html#plugins-filters-translate-dictionary_path


#3

Doesn't work when I remove the quotes.

I found the solution. This is my new dictionnary : "HOST@OID_interface": "Name interface"

I create a new configuration for logstash :

mutate {
    add_field =>  { "key_input_snmp" => "%{host}@%{[netflow][input_snmp]}" }
}

translate {
    dictionary_path => "/etc/logstash/yaml/interfaces.yaml"
    field => "key_input_snmp"
    destination => "tmp_input_interface"
    fallback => "unknown"
}

mutate {
    add_field => ["[netflow][input_interface]","%{[tmp_input_interface]}"]
}

It works !
Problem solved.

Thanks.


(Diego) #4

Thanks, could please share an example of interfaces.yaml

diego


(system) #5