Translate Filter plugin for a real CSV

Elastic Stack Logstash
1

Hello
I have a big CSV file(10 columns, 40Mb) that I want to use as a dictionary using the translate filter. I'm already doing that with a little csv files (2 columns, 4kb) and that is working fine. I'm doing it like this :

translate {
        field => "<numeric-value from beats, ie:101>"
        destination => "<name of the field I'll put the translation>"
        fallback => "not_found"
        dictionary => { #actually dictionaryPath
            "<numeric-value>" => "translation1" 
            "102" => "value2" 
            "190" => "value3" 
            }
        }

But I wanted to create a bigger dictionary with the other file, my idea is to assign [field] with a value I can look for in the CVS and get the dictionary looking what I want to translate based on the different columns from the CSV, not only a one-to-one translation...
is this possible somehow?

a similar thread I found is this... Translate filter plugin but that didn't gets to a solution.

thanks

2

You will need to have the second column of the CSV be a data format that another filter can process. For example:

  1. A csv with a different separator
  2. JSON
  3. key/value pairs

If the csv is static you can reformat it using logstash.

3

Thank you @Badger
hmmm, so like having the rest of the columns I want to read in one only column, and then process that, like:
dictionary => { #actually dictionaryPath
"" => "translation1"
"102" => "value2,value3,value4"
}
interesting... so, there is not way I can do this without reformatting the csv?

4

Not that I know of.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.