External flat file for host lookup to add tags on network syslogs


(Lee Carter) #1

All,

we use Rancid today to backup router/switch configurations. the config file in Rancid is the in format of:
dnsname;manufacture;status
so:

router01;cisco;up

I've figured out how to get syslog-ng to append the dnsname to the syslog line entry with a template $HOST in my destination configuration. now what I need to do is have logstash do some lookup where it references the above flat file and say... locate device-name, return device-type and add a tag of device type. this way I can do further parsing based on device types. does this make sense?

the biggest issue I have is how tell logstash to look at some external flat file and add tags appropriately.

thanks and sorry if similar questions like this have been asked already,

Lee


(Christian Dahlqvist) #2

If you can format you file with the additional data into YAML, you could use the translate filter plugin to enrich the records.


(Lee Carter) #3

Christian,

thanks... so I see:

filter {
  translate {
    dictionary => [ "100", "Continue",
                    "101", "Switching Protocols",
                    "merci", "thank you",
                    "old version", "new version" ]
  }
}

as an example... not sure I'm fully understanding how it works.. if it says every time you see "100" in a log entry replace 100 with "Continue"?

also, what if my "dictionary" were some external file? or would I need to configure that mapping directly into the logstash conf file?

thanks,

Lee


(Christian Dahlqvist) #4

You do not need to specify the data in the Logstash configuration file. The plugin allows you to specify a 'dictionary_path' to a file containing the data, which can be refreshed periodically. It also allows you to specify the destination field where to put the result of the lookup/translation. You can therefore look up a key in the external file and load a string into a new field, and then further process this if required.


(Lee Carter) #5

Ok... found a link where someone is using the translate field:

http://www.505forensics.com/do-you-even-bro-bro/

if I tune that just a bit:

    translate {
      # Hostname resolved by syslog-ng using the "template $HOST" in the destination field of syslog-ng.conf
      field => "syslog_hostname"
    
      # The field to "output" to - where do we want our lookup to go (never defined "device_type" so will that actually work) 
      destination => "device_type"
      # could I just "add_tag" as the destination somehow???
    
      # The dictionary to compare against. This can also be an external file
      dictionary => [ 
                    "devicea", "Cisco_Device",
                    "firewalla", "Cisco_ASA",
                    "firewallb", "PaloAlto",
                    "deviceb", "Arista",
                    ]
    }

if [device_type] =~ "Cisco_Device" {
    mutate {
        add_tag => "Cisco_Device"
    }
}

if [device_type =~ "Cisco_ASA" {
    mutate {
        add_tag => "Cisco_ASA"
    }
}

would something like this work?

thanks,

Lee


(Lee Carter) #6

Ok. So I finally got around to testing this and got it working with the following config:

> input {
>   stdin {
>      type => "STDIN-TEST"
>   }
> }
> filter {
>         # grok = break out source IP from message and replace message to be efficient
>         grok {
>          match => { "message" => "%{IPV4:device_ip} %{GREEDYDATA:message}" }
>          overwrite => [ "message" ]
>         }
>         # adding a field so we don't loose the source IP.. we can do reverse dns lookup now on "hostname"
>         mutate {
>          add_field => {
>           "hostname" => "%{device_ip}"
>          }
>         }
>         # revese lookup on hostname (was = IP... action replace sets it = fqdn
>         dns {
>          reverse => "hostname"
>          action => "replace"
>         }
>         # remove the domain suffix if you have a FQDN.. if IP then skip
>         if [hostname] !~ /(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/ {
>             grok {
>                 match => [ "[hostname]", "%{DATA:hostname}\.%{GREEDYDATA:domain}" ]
>                 overwrite => [ "hostname" ]
>             }
>         }
>         # do a lookup in a table to add field
>         translate {
>             field => "hostname"
>             destination => "devicetype"
>             dictionary_path => "/opt/elk/logstash/translate/devices.yaml"
>         }
>     }
> output {
>    stdout {
>         codec => rubydebug
>    }
> }


> devices.yaml:

> "deviceA": Cisco_Device
> "firewalla": Cisco_ASA
> "firewallb": PaloAlto
> "deviceb": Arista

I can now pass a sample message like:

10.1.1.1 this is a test message

grok will get the ip address (IPV4) and separate that from the original message. it then does a DNS lookup to get the FQDN.
finally we strip the domain stuff and pass the hostname to a flat file to return a device type.

thanks Christian for pointing me down this path. this is exactly what we need.

Note** for what it's worth I found a note stating that doing DNS resolution inside logstash isn't terribly efficient so we'll be using syslog-ng to resolve source IP to DNS and append that to the syslog file so for production we can skip the "dns & hostname suffix removal parts".


(system) #7