Filebeat and kubernetes with custom index

(Cristhian Saavedra) #1

Hi guys,
Im try to create a custom indexes for my different apps in a kubernetes cluster, but this not working :frowning:

anyone know any way to create this ?

Here is a part of my code

  hosts: ['${ELASTICSEARCH_HOST:x.x.x.x}:${ELASTICSEARCH_PORT:9200}']
  index: "${}-%{+yyyy.MM.dd}" 

Thanks in advance

(Steffen Siering) #2

The syntax for accessing fields is: %{[]}.

Assuming always exists the syntax is:

  index: "${[]}-%{+yyyy.MM.dd}" 

(Cristhian Saavedra) #3

I tried with your syntax, but is the same error

filebeat kubernetes Exiting: missing field accessing 'output.elasticsearch.index'

however I change

index: "%{[]}-%{+yyyy.MM.dd}" 

and add this lines "%{[]}"
setup.template.pattern: "%{[]}-*"

and don't show any error, but in elasticsearch don't show nothing

(Steffen Siering) #4

Filebeat normally provisions only one template mapping for one common index pattern. As you will end up with a many indices, you should disable index provisioning in filebeat (setting setup.template.enabled). Provisioning happens (create template mappings) on filebeat startup, it's no live operation. As is only available on events, you can not configure like this.

You might also reconsider the index names. E.g. filebeat creates index names like this: filebeat-<version>-<date of day>. Based on the index name filebeat creates a mapping template matching the filebeat version as well (so we don't get errors if schema changes between versions): filebeat-<version>-*.

The index names used by default in filebeat have a constant prefix, plus we can easily define one common mapping template. By using the strategy one can easily have filebeat manage the template mapping.

Also check the documentation of and setup.template.pattern. These settings always add the beat version to the pattern used by the template mapping.

Some alternative configs that might operate in the limits of filebeat + give you some more safety in case of schema changes: "k8s-logs"
setup.template.pattern: "k8s-logs-*"

  index: "k8s-logs-%{[beat.version]}-%{[]:dlq}-%{+yyyy.MM.dd}"

This configuration creates a common prefix "k8s-logs-" one can use in mapping templates and index patterns. If an event has no field named, then index uses the default name dlq (create pattern named k8s-logs-<beat version>-dlq-<date>), so you can monitor/reindex for incomplete/incorrect events.

It's somewhat unfortunate the is in the middle. As workaround you can use filebeat export template and change the mapping regex to "k8s-logs-*-<beat version>-*" in the json file. Use setup.template.json: settings to have filebeat load the modified template mapping:

setup.template.json.enabled: true
setup.template.json.path: "k8s-logs-template-<beat version>.json" "k8s-logs-template-<beat version>"

  index: "k8s-logs-%{[]:dlq}-%{[beat.version]}-%{+yyyy.MM.dd}"

show any error, but in elasticsearch don't show nothing

Without logs I can't really tell what the issue is.

(system) closed #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.