# External flat file for host lookup to add tags on network syslogs

**URL:** <https://discuss.elastic.co/t/external-flat-file-for-host-lookup-to-add-tags-on-network-syslogs/32579>\
**Category:** Logstash\
**Created:** [October 20, 2015, 2:49pm UTC](https://discuss.elastic.co/t/external-flat-file-for-host-lookup-to-add-tags-on-network-syslogs/32579 "2015-10-20T14:49:52Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![l1carter](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@l1carter](https://discuss.elastic.co/u/l1carter)\
**Post date:** [October 20, 2015, 2:49pm UTC](https://discuss.elastic.co/t/external-flat-file-for-host-lookup-to-add-tags-on-network-syslogs/32579/1 "2015-10-20T14:49:52Z")

</div>

All,

we use Rancid today to backup router/switch configurations. the config file in Rancid is the in format of:  
dnsname;manufacture;status  
so:

router01;cisco;up

I've figured out how to get syslog-ng to append the dnsname to the syslog line entry with a template $HOST in my destination configuration. now what I need to do is have logstash do some lookup where it references the above flat file and say... locate device-name, return device-type and add a tag of device type. this way I can do further parsing based on device types. does this make sense?

the biggest issue I have is how tell logstash to look at some external flat file and add tags appropriately.

thanks and sorry if similar questions like this have been asked already,

Lee

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 20, 2015, 2:53pm UTC](https://discuss.elastic.co/t/external-flat-file-for-host-lookup-to-add-tags-on-network-syslogs/32579/2 "2015-10-20T14:53:20Z")

</div>

If you can format you file with the additional data into YAML, you could use the [translate filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html) to enrich the records.

---

<div class="post-metadata">

**Author:** ![l1carter](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@l1carter](https://discuss.elastic.co/u/l1carter)\
**Post date:** [October 20, 2015, 3:29pm UTC](https://discuss.elastic.co/t/external-flat-file-for-host-lookup-to-add-tags-on-network-syslogs/32579/3 "2015-10-20T15:29:10Z")

</div>

Christian,

thanks... so I see:

```
filter {
  translate {
    dictionary => [ "100", "Continue",
                    "101", "Switching Protocols",
                    "merci", "thank you",
                    "old version", "new version" ]
  }
}

```

as an example... not sure I'm fully understanding how it works.. if it says every time you see "100" in a log entry replace 100 with "Continue"?

also, what if my "dictionary" were some external file? or would I need to configure that mapping directly into the logstash conf file?

thanks,

Lee

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 20, 2015, 4:07pm UTC](https://discuss.elastic.co/t/external-flat-file-for-host-lookup-to-add-tags-on-network-syslogs/32579/4 "2015-10-20T16:07:36Z")

</div>

You do not need to specify the data in the Logstash configuration file. The plugin allows you to specify a _'dictionary\_path_' to a file containing the data, which can be refreshed periodically. It also allows you to specify the destination field where to put the result of the lookup/translation. You can therefore look up a key in the external file and load a string into a new field, and then further process this if required.

---

<div class="post-metadata">

**Author:** ![l1carter](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@l1carter](https://discuss.elastic.co/u/l1carter)\
**Post date:** [October 20, 2015, 8:05pm UTC](https://discuss.elastic.co/t/external-flat-file-for-host-lookup-to-add-tags-on-network-syslogs/32579/5 "2015-10-20T20:05:16Z")

</div>

Ok... found a link where someone is using the translate field:

[http://www.505forensics.com/do-you-even-bro-bro/](http://www.505forensics.com/do-you-even-bro-bro/)

if I tune that just a bit:

```
    translate {
      # Hostname resolved by syslog-ng using the "template $HOST" in the destination field of syslog-ng.conf
      field => "syslog_hostname"
    
      # The field to "output" to - where do we want our lookup to go (never defined "device_type" so will that actually work) 
      destination => "device_type"
      # could I just "add_tag" as the destination somehow???
    
      # The dictionary to compare against. This can also be an external file
      dictionary => [ 
                    "devicea", "Cisco_Device",
                    "firewalla", "Cisco_ASA",
                    "firewallb", "PaloAlto",
                    "deviceb", "Arista",
                    ]
    }

if [device_type] =~ "Cisco_Device" {
    mutate {
        add_tag => "Cisco_Device"
    }
}

if [device_type =~ "Cisco_ASA" {
    mutate {
        add_tag => "Cisco_ASA"
    }
}

```

would something like this work?

thanks,

Lee

---

<div class="post-metadata">

**Author:** ![l1carter](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@l1carter](https://discuss.elastic.co/u/l1carter)\
**Post date:** [November 30, 2015, 4:04pm UTC](https://discuss.elastic.co/t/external-flat-file-for-host-lookup-to-add-tags-on-network-syslogs/32579/6 "2015-11-30T16:04:36Z")

</div>

Ok. So I finally got around to testing this and got it working with the following config:

```
> input {
> stdin {
> type => "STDIN-TEST"
> }
> }
> filter {
> # grok = break out source IP from message and replace message to be efficient
> grok {
> match => { "message" => "%{IPV4:device_ip} %{GREEDYDATA:message}" }
> overwrite => ["message"]
> }
> # adding a field so we don't loose the source IP.. we can do reverse dns lookup now on "hostname"
> mutate {
> add_field => {
> "hostname" => "%{device_ip}"
> }
> }
> # revese lookup on hostname (was = IP... action replace sets it = fqdn
> dns {
> reverse => "hostname"
> action => "replace"
> }
> # remove the domain suffix if you have a FQDN.. if IP then skip
> if [hostname] !~ /(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/ {
> grok {
> match => ["[hostname]", "%{DATA:hostname}\.%{GREEDYDATA:domain}" ]
> overwrite => ["hostname"]
> }
> }
> # do a lookup in a table to add field
> translate {
> field => "hostname"
> destination => "devicetype"
> dictionary_path => "/opt/elk/logstash/translate/devices.yaml"
> }
> }
> output {
> stdout {
> codec => rubydebug
> }
> }

> devices.yaml:

> "deviceA": Cisco_Device
> "firewalla": Cisco_ASA
> "firewallb": PaloAlto
> "deviceb": Arista

```

I can now pass a sample message like:

10.1.1.1 this is a test message

grok will get the ip address (IPV4) and separate that from the original message. it then does a DNS lookup to get the FQDN.  
finally we strip the domain stuff and pass the hostname to a flat file to return a device type.

thanks Christian for pointing me down this path. this is exactly what we need.

Note\*\* for what it's worth I found a note stating that doing DNS resolution inside logstash isn't terribly efficient so we'll be using syslog-ng to resolve source IP to DNS and append that to the syslog file so for production we can skip the "dns & hostname suffix removal parts".

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:20am UTC](https://discuss.elastic.co/t/external-flat-file-for-host-lookup-to-add-tags-on-network-syslogs/32579/7 "2017-07-06T05:20:46Z")

</div>