Logstash translate plugin dynamic dictionary path


I am creating a csv file which I want to use as a dictionary inside translate plugin.
Since the mapping might change the next day, I named my csv file this way:


where YYYY, MM, dd are year, month, date respectively.

I would like to read the latest csv file through the translate plugin. So, the logstash configuration used:

translate {
    field => "id"
    destination => "name"
    dictionary_path => "/home/my-mapping-%{+YYYY-MM-dd}.csv"

But, looks like dictionary_path parameter is unable to resolve %{+YYYY-MM-dd}
This is the error I got with it:

Invalid setting for translate filter plugin:
filter {
    translate {
      # This setting must be a path
      # File does not exist or cannot be opened /home/my-mapping-%{+YYYY-MM-dd}.csv
      dictionary_path => "/home/my-mapping-%{+YYYY-MM-dd}.csv"

Any workaround for this problem?


As a workaround, I am able to achieve this using Environment variables.

export DATE=$(date +%F)

Then inside config:

dictionary_path => "/home/my-mapping-${DATE}.csv"

Any better solutions are welcome.

Your solution obviously requires you to restart Logstash every day but apart from that it's probably the best available solution at this point.

Thanks for suggesting about restarting logstash. Didn't occur to mind.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.