So I'm using AWS Elasticsearch (which is currently at version 6.0.1). I use filebeat (6.1.0) to send logs to logstash (6.2.0), which uses the https://github.com/awslabs/logstash-output-amazon_es output plugin to write to an IAM protected ES cluster.
When I have no index template, documents get written to ES fine, but I get all those silly ".keyword" fields. So I found the docs on manually setting an index template (https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html)
I get the index template filebeat exports using:
filebeat export template --es.version=6.0.1 > filebeat-6.1.0.template.json
(Here's a gist with that template.json https://gist.github.com/djcrabhat/ce74ca10d74748a657f8f5c45c4654f1)
And I PUT that index template to ES. After that, I delete the filebeat-* indexes so they get recreated with the new template. The new index gets created, but even though I turned on debug logging and see logstash flushing data to ES, the index just sits there with 0 documents
GET /_cat/indices?v health status index uuid pri rep docs.count docs.deleted store.size pri.store.size [...] yellow open filebeat-6.1.0-2018.02.14 Zz8Jkf1XTyKxor-mL9fT-A 3 1 0 0 699b 699b
If I delete the index template, the logs start flowing to the index again
yellow open filebeat-6.1.0-2018.02.14 3aR0_cDHQIeYU6UR0CC0ww 5 1 6919 0 3.5mb 3.5mb
Any idea what's happening here? I'm testing this on a single node ES cluster while I take ELK for a spin.