I could use some help using metadata in file input plugin and elasticsearch output plugin.
I am trying to setup a single filter that provides configuration for the rest of my templates, for multiple test setups and production..
Here is what I start with (these are in different files, combined to show "relevant" parts).
filter {
mutate {
add_field => {"[@metadata][es_action]" => "index"}
add_field => {"[@metadata][es_index]" => "index_v1"}
add_field => {"[@metadata][ROOTDIR]" => "/Users/dev/logs/"}
add_field => {"[@metadata][HOSTS]" => "http://127.0.0.1:9200"}
}
}
input {
file {
path => [ "%{[@metadata][ROOTDIR]}/20180101.csv" ]
}
}
output {
stdout { codec => rubydebug { metadata => true } }
elasticsearch {
hosts => ["%{[@metadata][HOSTS]}" ]
action => "%{[@metadata][es_action]}"
index => "%{[@metadata][es_index]}"
}
}
The problems I have
- the file input plugin sees the path as relative "File paths must be absolute, relative path specified:"
- elasticsearch output plugin can't parse the URI "Malformed escape pair at index 0:"
but elasticsearch output plugin has no problem with
action => "%{[@metadata][es_action]}"
so URI parsing unexpanded string is a problem.
Maybe these fields can't handle metadata ?
I have tried many variations with no success.
Any suggestions are welcome.