Logstash with translate and external yaml file


(Lee Carter) #1

all,

I've got a logstash.conf file I'm working with and it's using a translate function like so:

    translate {
        field => "hostname"
        destination => "devtype"
        dictionary_path => "/opt/elk/logstash/translate/devmap.yaml"

I have a separate python script that runs nightly and checks rancid (our router/switch/firewall backup solution) and creates a yaml file appending newly added devices to rancid to this yaml dictionary.

in my testing if I append to the end of the yaml file while logstash is running it doesn't seem to recognize it... if I simply restart logstash the newly created line in the yaml file is now functional as expected...

does logstash load the contents of the yaml file to memory on startup and never reference it again? if so, i'm sort of wondering other than config file size why using a file versus hard coding into the config is any better...

thanks,

Lee


(Christian Dahlqvist) #2

The 'refresh_interval' parameter controls how often the yams file is reloaded. The default value seems to be 5 minutes. How long have you waited before you validate that the file has been refreshed and is affecting the output?


(Magnus B├Ąck) #3

I note that the reload code looks like this:

What happens if the updated file deletes entries? It looks like merge!() only adds keys from the newly loaded YAML.


(Lee Carter) #4

no time at all. just tried after I appended. I'll give it some time and see if it just works.

thanks.


(Lee Carter) #5

good point. technically we were going to simply re-crate the file nightly. I know I said append but that's how I'm doing it in testing... just echo new values to the end of the file for quick tests. in prod however the plan would be to create a new authoritative list nightly since we do remove devices from the network from time to time :slight_smile:

guess I'll need to test how it does with a merge if it reflects removed devices properly.


(Lee Carter) #8

that was it. just a little time was all it needed. thanks.


(system) #9