Filebeat and kubernetes with custom index

[CSaavedra (Cristhian Saavedra)]

Hi guys,
Im try to create a custom indexes for my different apps in a kubernetes cluster, but this not working

anyone know any way to create this ?

Here is a part of my code

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:x.x.x.x}:${ELASTICSEARCH_PORT:9200}']
  index: "${data.kubernetes.labels.app}-%{+yyyy.MM.dd}" 

Thanks in advance

[steffens (Steffen Siering)]

The syntax for accessing fields is: %{[field.name]}.

Assuming data.kubernetes.labels.app always exists the syntax is:

  index: "${[data.kubernetes.labels.app]}-%{+yyyy.MM.dd}" 

[CSaavedra (Cristhian Saavedra)]

I tried with your syntax, but is the same error

filebeat kubernetes Exiting: missing field accessing 'output.elasticsearch.index'

however I change

index: "%{[data.kubernetes.labels.app]}-%{+yyyy.MM.dd}" 

and add this lines

setup.template.name: "%{[data.kubernetes.labels.app]}"
setup.template.pattern: "%{[data.kubernetes.labels.app]}-*"

and don't show any error, but in elasticsearch don't show nothing

[steffens (Steffen Siering)]

Filebeat normally provisions only one template mapping for one common index pattern. As you will end up with a many indices, you should disable index provisioning in filebeat (setting setup.template.enabled). Provisioning happens (create template mappings) on filebeat startup, it's no live operation. As data.kubernetes.labels.app is only available on events, you can not configure setup.template.name like this.

You might also reconsider the index names. E.g. filebeat creates index names like this: filebeat-<version>-<date of day>. Based on the index name filebeat creates a mapping template matching the filebeat version as well (so we don't get errors if schema changes between versions): filebeat-<version>-*.

The index names used by default in filebeat have a constant prefix, plus we can easily define one common mapping template. By using the strategy one can easily have filebeat manage the template mapping.

Also check the documentation of setup.template.name and setup.template.pattern. These settings always add the beat version to the pattern used by the template mapping.

Some alternative configs that might operate in the limits of filebeat + give you some more safety in case of schema changes:

setup.template.name: "k8s-logs"
setup.template.pattern: "k8s-logs-*"

output.elasticsearch:
  index: "k8s-logs-%{[beat.version]}-%{[data.kubernetes.labels.app]:dlq}-%{+yyyy.MM.dd}"

This configuration creates a common prefix "k8s-logs-" one can use in mapping templates and index patterns. If an event has no field named data.kubernetes.labels.app, then index uses the default name dlq (create pattern named k8s-logs-<beat version>-dlq-<date>), so you can monitor/reindex for incomplete/incorrect events.

It's somewhat unfortunate the is in the middle. As workaround you can use filebeat export template and change the mapping regex to "k8s-logs-*-<beat version>-*" in the json file. Use setup.template.json: settings to have filebeat load the modified template mapping:

setup.template.json.enabled: true
setup.template.json.path: "k8s-logs-template-<beat version>.json"
setup.template.json.name: "k8s-logs-template-<beat version>"

output.elasticsearch:
  index: "k8s-logs-%{[data.kubernetes.labels.app]:dlq}-%{[beat.version]}-%{+yyyy.MM.dd}"

show any error, but in elasticsearch don't show nothing

Without logs I can't really tell what the issue is.