Translate filter not working properly

[cskowronnek (Conrad)]

Hi,

I'm using Logstash 6.1.1 to parse some Apache logs and ship them to ES.

I use the translate filter in my config to translate software IDs to filenames.

Logstash config:

if [request] =~ /sdc/ and [product][softwareId] =~ /.+/ {
                    mutate {
                        add_field => {
                            "[translate_in]" => "%{[product][softwareId]} PR"
                        }
                    }
                    translate {
                        dictionary_path => "/etc/logstash/sc_product_sw.yml"
                        destination => "[translate_out]"
                        field => "[translate_in]"
                        fallback => "No filename found for %{[product][softwareId]} PR"
                    }
                    mutate {
                        add_tag => [ "product_download", "sc_product_download" ]
                        add_field => {
                            "[product][downloaded_file]" => "%{[translate_out]}"
                        }
                        remove_field => [
                            "[translate_in]",
                            "[translate_out]"
                        ]
                    }

Example from sc_product_sw.yml:

"10301": File Name 123.zip
"10909": C12345.zip
"11063": Setup_XYZ.zip
"11322": 2132131_ABC_099931.zio
"11443": Flash Patch v123.00 Setup.exe
"487": X500 DVD.zip
"505": Explore Infinity XX1.0.zip
"1901": ABC_DEFG_ADSA_ASDDD.zip

The problem is, logstash is resolving most of the IDs to filenames, but not all. Some values aren't mapped although there are matching IDs in the file.

Has anyone an idea how to approach this problem?
Thanks in advance.

[magnusbaeck (Magnus Bäck)]

Please give an example of a document where you're not getting a match. Use a stdout { codec => rubydebug } output to dump the raw event and (temporarily) remove the mutate filter that destroys the evidence.

[cskowronnek (Conrad)]

Hi Magnus, thank you for your reply. After restarting logstash and reindexing all the logs through filebeat I got all filenames resolved. Maybe there was a problem with the auto refresh of the yaml dictionaries. I'll keep an eye on this. So from my current understanding there is no problem with the translate filter.