I'm using Logstash 5.5.1 and ElasticSearch 5.6.4. I'm ingesting Filebeat log data, and I'm seeing a very strange result when using the logstash-filter-clone. The intended indexes (app_stats-2017.11.17) are missing, but I discovered literal logstash template vars being used for the index names.
GET /_cat/indices
green open %{[@metadata][type]}-2017.11.14 d-pB3vUgSXyG8TdFt7BO-w 5 1 1105 0 1.6mb 854.8kb
green open app-2017.11.06 CgUNInkETQCy0KMNEcqSYw 5 1 205 0 1mb 474kb
green open %{[@metadata][type]}-2017.11.18 bbj0cqsDRfqz-cTF7b7rRA 5 1 73680 0 42.8mb 21.4mb
green open app-2017.11.18 -WgpIx9-Rqq2l2a9Hgn2AQ 5 1 1266405 0 1gb 525.2mb
green open app-2017.11.17 Me4iKTEFSZ-aKUQs58PaDw 5 1 7428668 0 7gb 3.7gb
green open app-2017.11.14 sPc-3mMQTZ-m-JROmo9b8A 5 1 67800 0 73.3mb 36.7mb
Here is the clone, mutate, and output of the only Logstash Pipeline config being used:
[...]
if [cache_stats] {
# (Clone some fields to a second index, with just stats/metrics)
clone {
clones => ["app_stats"]
remove_field => ["ppid", "pid", "message", "queue", "level"]
}
}
### App Statistics Index ###
if [type] == "app_stats" {
mutate {
update => { "[@metadata][type]" => "app_stats" }
}
}
}
output {
# fields.format is set by Filebeat
if [fields][format] == "json" {
elasticsearch {
hosts => ["elasticsearch:9200"]
codec => json
manage_template => false
index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else {
elasticsearch {
hosts => ["elasticsearch:9200"]
manage_template => false
index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
I've been unable to find any threads, Github issues, or search results any way related to this, but it's a difficult issue to search for as well. Any ideas?