Using metadata in file input plugin and elasticsearch output plugin

I could use some help using metadata in file input plugin and elasticsearch output plugin.

I am trying to setup a single filter that provides configuration for the rest of my templates, for multiple test setups and production..

Here is what I start with (these are in different files, combined to show "relevant" parts).

filter {
mutate {
add_field => {"[@metadata][es_action]" => "index"}
add_field => {"[@metadata][es_index]" => "index_v1"}
add_field => {"[@metadata][ROOTDIR]" => "/Users/dev/logs/"}
add_field => {"[@metadata][HOSTS]" => ""}

input {
file {
path => [ "%{[@metadata][ROOTDIR]}/20180101.csv" ]

output {
stdout { codec => rubydebug { metadata => true } }
elasticsearch {
hosts => ["%{[@metadata][HOSTS]}" ]
action => "%{[@metadata][es_action]}"
index => "%{[@metadata][es_index]}"

The problems I have

  1. the file input plugin sees the path as relative "File paths must be absolute, relative path specified:"
  2. elasticsearch output plugin can't parse the URI "Malformed escape pair at index 0:"

but elasticsearch output plugin has no problem with
action => "%{[@metadata][es_action]}"
so URI parsing unexpanded string is a problem.

Maybe these fields can't handle metadata ?

I have tried many variations with no success.

Any suggestions are welcome.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.