Filebeat normally provisions only one template mapping for one common index pattern. As you will end up with a many indices, you should disable index provisioning in filebeat (setting setup.template.enabled
). Provisioning happens (create template mappings) on filebeat startup, it's no live operation. As data.kubernetes.labels.app
is only available on events, you can not configure setup.template.name
like this.
You might also reconsider the index names. E.g. filebeat creates index names like this: filebeat-<version>-<date of day>
. Based on the index name filebeat creates a mapping template matching the filebeat version as well (so we don't get errors if schema changes between versions): filebeat-<version>-*
.
The index names used by default in filebeat have a constant prefix, plus we can easily define one common mapping template. By using the strategy one can easily have filebeat manage the template mapping.
Also check the documentation of setup.template.name and setup.template.pattern. These settings always add the beat version to the pattern used by the template mapping.
Some alternative configs that might operate in the limits of filebeat + give you some more safety in case of schema changes:
setup.template.name: "k8s-logs"
setup.template.pattern: "k8s-logs-*"
output.elasticsearch:
index: "k8s-logs-%{[beat.version]}-%{[data.kubernetes.labels.app]:dlq}-%{+yyyy.MM.dd}"
This configuration creates a common prefix "k8s-logs-" one can use in mapping templates and index patterns. If an event has no field named data.kubernetes.labels.app
, then index
uses the default name dlq
(create pattern named k8s-logs-<beat version>-dlq-<date>
), so you can monitor/reindex for incomplete/incorrect events.
It's somewhat unfortunate the is in the middle. As workaround you can use filebeat export template
and change the mapping regex to "k8s-logs-*-<beat version>-*"
in the json file. Use setup.template.json:
settings to have filebeat load the modified template mapping:
setup.template.json.enabled: true
setup.template.json.path: "k8s-logs-template-<beat version>.json"
setup.template.json.name: "k8s-logs-template-<beat version>"
output.elasticsearch:
index: "k8s-logs-%{[data.kubernetes.labels.app]:dlq}-%{[beat.version]}-%{+yyyy.MM.dd}"
show any error, but in elasticsearch don't show nothing
Without logs I can't really tell what the issue is.