External flat file for host lookup to add tags on network syslogs

Elastic Stack Logstash
1

All,

we use Rancid today to backup router/switch configurations. the config file in Rancid is the in format of:
dnsname;manufacture;status
so:

router01;cisco;up

I've figured out how to get syslog-ng to append the dnsname to the syslog line entry with a template $HOST in my destination configuration. now what I need to do is have logstash do some lookup where it references the above flat file and say... locate device-name, return device-type and add a tag of device type. this way I can do further parsing based on device types. does this make sense?

the biggest issue I have is how tell logstash to look at some external flat file and add tags appropriately.

thanks and sorry if similar questions like this have been asked already,

Lee

2

If you can format you file with the additional data into YAML, you could use the translate filter plugin to enrich the records.

1 Like
3

Christian,

thanks... so I see:

filter {
  translate {
    dictionary => [ "100", "Continue",
                    "101", "Switching Protocols",
                    "merci", "thank you",
                    "old version", "new version" ]
  }
}

as an example... not sure I'm fully understanding how it works.. if it says every time you see "100" in a log entry replace 100 with "Continue"?

also, what if my "dictionary" were some external file? or would I need to configure that mapping directly into the logstash conf file?

thanks,

Lee

4

You do not need to specify the data in the Logstash configuration file. The plugin allows you to specify a 'dictionary_path' to a file containing the data, which can be refreshed periodically. It also allows you to specify the destination field where to put the result of the lookup/translation. You can therefore look up a key in the external file and load a string into a new field, and then further process this if required.

1 Like
5

Ok... found a link where someone is using the translate field:

http://www.505forensics.com/do-you-even-bro-bro/

if I tune that just a bit:

    translate {
      # Hostname resolved by syslog-ng using the "template $HOST" in the destination field of syslog-ng.conf
      field => "syslog_hostname"
    
      # The field to "output" to - where do we want our lookup to go (never defined "device_type" so will that actually work) 
      destination => "device_type"
      # could I just "add_tag" as the destination somehow???
    
      # The dictionary to compare against. This can also be an external file
      dictionary => [ 
                    "devicea", "Cisco_Device",
                    "firewalla", "Cisco_ASA",
                    "firewallb", "PaloAlto",
                    "deviceb", "Arista",
                    ]
    }

if [device_type] =~ "Cisco_Device" {
    mutate {
        add_tag => "Cisco_Device"
    }
}

if [device_type =~ "Cisco_ASA" {
    mutate {
        add_tag => "Cisco_ASA"
    }
}

would something like this work?

thanks,

Lee

6

Ok. So I finally got around to testing this and got it working with the following config:

> input {
>   stdin {
>      type => "STDIN-TEST"
>   }
> }
> filter {
>         # grok = break out source IP from message and replace message to be efficient
>         grok {
>          match => { "message" => "%{IPV4:device_ip} %{GREEDYDATA:message}" }
>          overwrite => [ "message" ]
>         }
>         # adding a field so we don't loose the source IP.. we can do reverse dns lookup now on "hostname"
>         mutate {
>          add_field => {
>           "hostname" => "%{device_ip}"
>          }
>         }
>         # revese lookup on hostname (was = IP... action replace sets it = fqdn
>         dns {
>          reverse => "hostname"
>          action => "replace"
>         }
>         # remove the domain suffix if you have a FQDN.. if IP then skip
>         if [hostname] !~ /(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/ {
>             grok {
>                 match => [ "[hostname]", "%{DATA:hostname}\.%{GREEDYDATA:domain}" ]
>                 overwrite => [ "hostname" ]
>             }
>         }
>         # do a lookup in a table to add field
>         translate {
>             field => "hostname"
>             destination => "devicetype"
>             dictionary_path => "/opt/elk/logstash/translate/devices.yaml"
>         }
>     }
> output {
>    stdout {
>         codec => rubydebug
>    }
> }


> devices.yaml:

> "deviceA": Cisco_Device
> "firewalla": Cisco_ASA
> "firewallb": PaloAlto
> "deviceb": Arista

I can now pass a sample message like:

10.1.1.1 this is a test message

grok will get the ip address (IPV4) and separate that from the original message. it then does a DNS lookup to get the FQDN.
finally we strip the domain stuff and pass the hostname to a flat file to return a device type.

thanks Christian for pointing me down this path. this is exactly what we need.

Note** for what it's worth I found a note stating that doing DNS resolution inside logstash isn't terribly efficient so we'll be using syslog-ng to resolve source IP to DNS and append that to the syslog file so for production we can skip the "dns & hostname suffix removal parts".

1 Like