Dynamic index name with template setup fails on using an existing ILM

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi guys,

I am currently trying to define a ILM that should be used for a specific template.
However, I am failing unfortunately.

The manifest looks like that:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 7.8.1
  elasticsearchRef:
    name: elastic
    namespace: elastic-system
    # Currently this setting requires Kibana to have TLS enabled.
    # See https://github.com/elastic/cloud-on-k8s/issues/3523 for more information.
  kibanaRef:
    name: kibana
    namespace: elastic-system
  config:
    output.elasticsearch.index: "%{[kubernetes.namespace]}-filebeat-%{+xxxx.ww}"
    output.elasticsearch.username: "filebeat"
    output.elasticsearch.password: "xyz"
    setup.template.name: "filebeat"
    setup.template.pattern: "*"
    setup.template.overwrite: true
    setup.template.settings:
      index.number_of_shards: 3
      index.number_of_replicas: 1
    setup.ilm.enabled: false
    setup.ilm.policy_name: "denic-index-policy"
    setup.dashboards.enabled: true

As you can see the template filebeat will be create / overwritten everytime I deploy this manifest. With that a pattern selector for all indicies will be set. What I am failing now is to set the ILM that this template should use. I have a self build ILM in place, that should be attached to this template. However, it seems like setup.ilm.enabled: false disables the whole ILM usage for the template. I already tried using auto and enabled. When using them, the ILM gets set but output.elasticsearch.index will be ignored.

Any idea?

2

Hi @raulgs, thanks for your question.

I think that is the expected behaviour - see Index lifecycle management:

If index lifecycle management is enabled (which is typically the default), setup.template.name and setup.template.pattern are ignored.

and filebeat-reference.yml:

# Configure index lifecycle management (ILM). These settings create a write
# alias and add additional settings to the index template. When ILM is enabled,
# output.elasticsearch.index is ignored, and the write alias is used to set the
# index name.

As to the why this is the case or if you have any other questions on this particular subject, the Beats forum would be the best place to ask.

3

Alright I just figured out that this works as well

    setup.template.enabled: true
    setup.template.name: "filebeat-%{[agent.version]}"
    setup.template.pattern: "*-filebeat-*"
    setup.template.order: 50
    setup.template.overwrite: true
    setup.template.settings:
      index.number_of_shards: 3
      index.number_of_replicas: 1
      index.lifecycle.name: "denic-index-policy"

So I am fine. Thanks :slight_smile:

4

That's great to hear, thanks for the update.