How to create indices based on kubernetes metadata

Elastic Stack Beats
1

I am using filebeat in kubernetes to ship logs to Elasticsearch.
I want to create indexes based on namespaces . I tried this :

output.elasticsearch:
hosts: ["http://localhost:9200"]
index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
indices:
- index: "namespace1-%{[beat.version]}-%{+yyyy.MM.dd}"
when.contains:
kubernetes.namepsace: "namespace1"
- index: "namespace2-%{[beat.version]}-%{+yyyy.MM.dd}"
when.contains:
kubernetes.namepsace: "namespace2"

But it is not creating those indexes.
Can anyone please help me?

2

Hi @Rohit_Vyavahare,

Please use preformatted text when pasting confs, indenting is lost if not and we cannot really troubleshoot the issue.

From what I can tell, you can have some things into account:

  • There is a type, you are writing kubernetes.namepsace, notice the namepsace.
  • You can use variables in the index setting, so this could be:
index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
indices:
  - index: "%{[[kubernetes.namespace]]}-%{[beat.version]}-%{+yyyy.MM.dd}"
    when.regexp:
     kubernetes.namepsace: ".*"

Best regards

3

Please format logs, configs and terminal input/output using the </>-Button or markdown code fences. This forum uses Markdown to format posts. Without proper formatting, it can be very hard to read your posts.

Config files using YAML are sensitive to formatting and indentation. Without proper formatting it is difficult to spot any errors in your configs.

You get any errors in filebeat debug log? Do you find events with kubernetes.namespace: namespace1 in the filebeat index?

Given the namespace name doesn't change this should work better:

output.elasticsearch:
  index: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"
1 Like
4

Thank you @steffens , this solved my problem. My apologies for question formating.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.