Replace a value to another value logstash

[mikhatanu (Mikhael)]

Hello, i have a log with an id. In another file, i have list file of the id mapping. for example id 1 has a name, description, etc in the list file. I want to replace the id in that log to the data in the list file.

Log example:

id, ip_source, ip_dest
23, x.x.x.x, y.y.y.y

file list example:

id, name, description
23, web_app, "this is a web app"

Output to elasticsearch:

23, x.x.x.x, y.y.y.y, web_app, "this is a web app"

can i do this all in logstash? or should i make them into their own indices in elasticsearch and filter it in kibana?

[leandrojmp (Leandro Pereira)]

You can do it in logstash using the translate filter, the main issue is that the translate filter supports only key-value pairs, so for each key it haves just one value.

Since you have two values, name and description, you can use two files as the dictionary for the translate filters, but you can also use just one file and parse the value later.

For the second case, and considering that you are already parsing your message, your dictionary file would be something like this:

dictionary.yml

"23": "webapp;this is an webapp"

Then you will need the following filters:

translate {
	source => "id"
	target => "[@metadata][translated]"
	dictionary_path => "/path/to/the/file/dictionary.yml"
	refresh_interval => 300
}

When the field id exists as a key in the file dictionary.yml, this filter will create the field [@metadata][translated] with the value from the file, which will be web_app;this is an web app
, now you can parse this field using dissect to extract those values in different fields.

dissect {
    mapping => {
        "[@metadata][translated]" => "%{name};%{description}"
    }
}

You do not need to worry about the [@metadata][translated] in your documents, this is a temporary field that will not be present in your output.

If you want a longer explanation about the translate filter, I've made a blog post about it a couple of time ago.