Dynamic dictionary?

alex3 · July 5, 2018, 9:10pm

For some reason I can't reference a field in my dictionary_path, is this even possible? I want to use a different dictionary depending on a field.

warkolm · July 5, 2018, 9:27pm

Please show your config, and share what version you are running.

alex3 · July 5, 2018, 9:30pm

I am using 6.2.4 and here's the translation portion of my config file:

translate { 
    field => "metricId"
    dictionary_path => "/Users/user/diagviz/dict/%{[bundle]}.json"
    destination => "metricTemp"
    override => true 

}

Where bundle I know is a field that is defined, but the output doesn't replace it with the field when I get the error message.

This is the error message:
[2018-07-05T14:31:57,713][ERROR][logstash.filters.translate] Invalid setting for translate filter plugin:

filter {
translate {
# This setting must be a path
# File does not exist or cannot be opened /Users/user/diagviz/dict/%{[bundle]}.json
dictionary_path => "/Users/user/diagviz/dict/%{[bundle]}.json"
...
}
}

alex3 · July 5, 2018, 10:00pm

Environment variables do work though.

Shaoranlaos · July 6, 2018, 6:10am

With a quick view in the source code it seems that is not possible

github.com

logstash-plugins/logstash-filter-translate/blob/1e056f4f4872eca29307bf05a5b1d3b8fff881d3/lib/logstash/filters/translate.rb#L253


  end
rescue => e
  loading_exception(e, raise_exception)
end


def load_yaml(raise_exception=false)
  if !File.exists?(@dictionary_path)
    @logger.warn("dictionary file read failure, continuing with old dictionary", :path => @dictionary_path)
    return
  end
  refresh_dictionary!(YAML.load_file(@dictionary_path))
end


def load_json(raise_exception=false)
  if !File.exists?(@dictionary_path)
    @logger.warn("dictionary file read failure, continuing with old dictionary", :path => @dictionary_path)
    return
  end
  refresh_dictionary!(JSON.parse(File.read(@dictionary_path)))
end

Christian_Dahlqvist · July 6, 2018, 6:58am

I suspect you may need to use one translate plugin per dictionary and control which one you use through conditionals. You might also be able to create a single dictionary where the key contains the bundle, so you can have all data in one file.

alex3 · July 6, 2018, 4:54pm

The thing is my dictionaries will be variable (I won't know how many dictionaries I need so I generate them dynamically)

Shaoranlaos · July 10, 2018, 5:32am

eventualy you could try instead of the translate filter to use the jdbc static or streaming filter?

https://www.elastic.co/guide/en/logstash/current/plugins-filters-jdbc_static.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-jdbc_streaming.html

using these with a sqlite db as source would probaly fill your needs

system · August 7, 2018, 5:32am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.