# Filebeat and kubernetes with custom index

**URL:** <https://discuss.elastic.co/t/filebeat-and-kubernetes-with-custom-index/156336>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 12, 2018, 6:51pm UTC](https://discuss.elastic.co/t/filebeat-and-kubernetes-with-custom-index/156336 "2018-11-12T18:51:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![CSaavedra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/csaavedra/32/37550_2.png) [@CSaavedra](https://discuss.elastic.co/u/CSaavedra)\
**Post date:** [November 12, 2018, 6:51pm UTC](https://discuss.elastic.co/t/filebeat-and-kubernetes-with-custom-index/156336/1 "2018-11-12T18:51:26Z")

</div>

Hi guys,  
Im try to create a custom indexes for my different apps in a kubernetes cluster, but this not working ☹

anyone know any way to create this ?

Here is a part of my code

```
output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:x.x.x.x}:${ELASTICSEARCH_PORT:9200}']
  index: "${data.kubernetes.labels.app}-%{+yyyy.MM.dd}" 

```

Thanks in advance

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 12, 2018, 9:02pm UTC](https://discuss.elastic.co/t/filebeat-and-kubernetes-with-custom-index/156336/2 "2018-11-12T21:02:08Z")

</div>

The syntax for accessing fields is: `%{[field.name]}`.

Assuming `data.kubernetes.labels.app` always exists the syntax is:

```auto
  index: "${[data.kubernetes.labels.app]}-%{+yyyy.MM.dd}" 

```

---

<div class="post-metadata">

**Author:** ![CSaavedra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/csaavedra/32/37550_2.png) [@CSaavedra](https://discuss.elastic.co/u/CSaavedra)\
**Post date:** [November 13, 2018, 2:21pm UTC](https://discuss.elastic.co/t/filebeat-and-kubernetes-with-custom-index/156336/3 "2018-11-13T14:21:18Z")

</div>

I tried with your syntax, but is the same error

```
filebeat kubernetes Exiting: missing field accessing 'output.elasticsearch.index'

```

however I change

```
index: "%{[data.kubernetes.labels.app]}-%{+yyyy.MM.dd}" 

```

and add this lines

```
setup.template.name: "%{[data.kubernetes.labels.app]}"
setup.template.pattern: "%{[data.kubernetes.labels.app]}-*"

```

and don't show any error, but in elasticsearch don't show nothing

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 14, 2018, 3:08pm UTC](https://discuss.elastic.co/t/filebeat-and-kubernetes-with-custom-index/156336/4 "2018-11-14T15:08:32Z")

</div>

Filebeat normally provisions only one template mapping for one common index pattern. As you will end up with a many indices, you should disable index provisioning in filebeat (setting `setup.template.enabled`). Provisioning happens (create template mappings) on filebeat startup, it's no live operation. As `data.kubernetes.labels.app` is only available on events, you can not configure `setup.template.name` like this.

You might also reconsider the index names. E.g. filebeat creates index names like this: `filebeat-<version>-<date of day>`. Based on the index name filebeat creates a mapping template matching the filebeat version as well (so we don't get errors if schema changes between versions): `filebeat-<version>-*`.

The index names used by default in filebeat have a constant prefix, plus we can easily define one common mapping template. By using the strategy one can easily have filebeat manage the template mapping.

Also check the documentation of [setup.template.name and setup.template.pattern](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html#configuration-template). These settings always add the beat version to the pattern used by the template mapping.

Some alternative configs that might operate in the limits of filebeat + give you some more safety in case of schema changes:

```auto
setup.template.name: "k8s-logs"
setup.template.pattern: "k8s-logs-*"

output.elasticsearch:
  index: "k8s-logs-%{[beat.version]}-%{[data.kubernetes.labels.app]:dlq}-%{+yyyy.MM.dd}"

```

This configuration creates a common prefix "k8s-logs-" one can use in mapping templates and index patterns. If an event has no field named `data.kubernetes.labels.app`, then `index` uses the default name `dlq` (create pattern named `k8s-logs-<beat version>-dlq-<date>`), so you can monitor/reindex for incomplete/incorrect events.

It's somewhat unfortunate the is in the middle. As workaround you can use `filebeat export template` and change the mapping regex to `"k8s-logs-*-<beat version>-*"` in the json file. Use `setup.template.json:` settings to have filebeat load the modified template mapping:

```auto
setup.template.json.enabled: true
setup.template.json.path: "k8s-logs-template-<beat version>.json"
setup.template.json.name: "k8s-logs-template-<beat version>"

output.elasticsearch:
  index: "k8s-logs-%{[data.kubernetes.labels.app]:dlq}-%{[beat.version]}-%{+yyyy.MM.dd}"

```

> show any error, but in elasticsearch don't show nothing

Without logs I can't really tell what the issue is.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 12, 2018, 3:08pm UTC](https://discuss.elastic.co/t/filebeat-and-kubernetes-with-custom-index/156336/5 "2018-12-12T15:08:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.