Logstash clone resulting in literal "%{[@metadata][type]}" index names!

I'm using Logstash 5.5.1 and ElasticSearch 5.6.4. I'm ingesting Filebeat log data, and I'm seeing a very strange result when using the logstash-filter-clone. The intended indexes (app_stats-2017.11.17) are missing, but I discovered literal logstash template vars being used for the index names.

GET /_cat/indices

green open %{[@metadata][type]}-2017.11.14 d-pB3vUgSXyG8TdFt7BO-w 5 1    1105  0   1.6mb 854.8kb
green open app-2017.11.06              CgUNInkETQCy0KMNEcqSYw 5 1     205  0     1mb   474kb
green open %{[@metadata][type]}-2017.11.18 bbj0cqsDRfqz-cTF7b7rRA 5 1   73680  0  42.8mb  21.4mb
green open app-2017.11.18              -WgpIx9-Rqq2l2a9Hgn2AQ 5 1 1266405  0     1gb 525.2mb
green open app-2017.11.17              Me4iKTEFSZ-aKUQs58PaDw 5 1 7428668  0     7gb   3.7gb
green open app-2017.11.14              sPc-3mMQTZ-m-JROmo9b8A 5 1   67800  0  73.3mb  36.7mb

Here is the clone, mutate, and output of the only Logstash Pipeline config being used:

  [...]
  if [cache_stats] { 
    # (Clone some fields to a second index, with just stats/metrics)
    clone {
      clones => ["app_stats"]
      remove_field => ["ppid", "pid", "message", "queue", "level"]
    }
  }

  ### App Statistics Index ###
  if [type] == "app_stats" {
    mutate {
      update => { "[@metadata][type]" => "app_stats" }
    }
  }
}

output {
  # fields.format is set by Filebeat
  if [fields][format] ==  "json" {
     elasticsearch {
        hosts => ["elasticsearch:9200"]
        codec => json
        manage_template => false
        index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
     }
  } else {
     elasticsearch {
        hosts => ["elasticsearch:9200"]
        manage_template => false
        index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
     }
  }

I've been unable to find any threads, Github issues, or search results any way related to this, but it's a difficult issue to search for as well. Any ideas?

Ok, in the process of writing this, I discovered that our installation of Logstash is a bit behind. Updating LS to 5.6.4 actually resolved this problem!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.