Programmatically configure ILM

Elastic Stack Elasticsearch
1

Please forgive me if this specific topic has already been discussed elsewhere, I couldn't find much information matching what I'm looking for.

I'm managing EFK stacks across multiple Kubernetes clusters and while the deployment of the various EFK components is fully automated in my environment using Helm charts and our CI/CD pipeline, many of the configuration aspects of Elasticsearch and Kibana I'm having difficulty managing.

One of those difficulties is around ILM and programmatically configuring the index template. While I can create and manage this using the API, it can be cumbersome in our environment to have to go through these manual steps for each new Kubernetes cluster we create or recreate. Is there currently any way or any plans in the future to allow this to be done in an automated fashion?

I'm thinking similar to how certain attributes can be defined in the elasticsearch.yml and log4j2.properties files which we're able to define in our Helm chart values.yml files for Elasticsearch. By allowing for this, each time a new Kubernetes cluster is spun up, the index template we have defined whill in turn be created on each Elasticsearch node and match across the entire Elasticsearch cluster.

If I'm missing something or not understanding any reason that this can't be done, I'm all ears. If anybody has done something to automate this themselves such as configuring a CronJob to make the API call to create those index templates upon cluster creation that would be great as well though I'm a bit junior in my skills in this area.

Thanks!

1 Like
2

It sounds like this may be something that should be handled by the kubernetes operator, to do any setup automatically when a cluster is created.

I don't know of any work to add this, but since the k8s operator is relatively new, you should open an issue with what you'd like to do at GitHub - elastic/cloud-on-k8s: Elastic Cloud on Kubernetes

3

Thanks @dakrone. I've checked out Elastic Cloud on K8s but it's been some time since I've looked. For now I ended up deploying the elasticsearch-curator to perform our index cleanup and rollup jobs. I had previously given up on the curator as the helm charts reference a docker hub that hasn't had an image update in over a year and the last image isn't compatible with Elasticsearch 7.x that we have deployed but I found a different image repository for the curator that is updated and I now have it deployed and testing.

I think the existing method of deploying Elasticsearch via Helm charts isn't the greatest due to many settings not supporting any programmatic way of configuring along with the Helm chart deployment and instead would require development of additional tooling to make API calls but thank you again for linking Elastic Cloud on K8s, it may be what we need moving forward.

4

I know you said you are using EFK, but the way we handle this for our clusters is to configure the ILM policy within filebeat/metricbeat. If you are using the official helm charts the configuration looks something like:

filebeatConfig:
  filebeat.yml: |
    filebeat.autodiscover:
      providers:
      - type: kubernetes
        hints.enabled: true
        templates:
          - condition:
              equals:
                kubernetes.labels.logtype: json
            config:
              - type: docker
                containers.ids:
                  - "$${data.kubernetes.container.id}"
                document_type: kube-logs
                fields:
                  host: ${FILEBEAT_HOST:${HOSTNAME}}
                fields_under_root: true
                json.message_key: message

    output.elasticsearch:
      hosts: ["${ELASTICSEARCH_HOST}"]
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"

    setup.ilm:
      enabled: true
      overwrite: true
      policy_file: /usr/share/filebeat/ilm.json

  ilm.json: |
    {
      "policy": {
        "phases": {
          "hot": {
            "actions": {
              "rollover": {
                "max_age": "1d"
              }
            }
          },
          "delete": {
            "min_age": "30d",
            "actions": {
              "delete": {}
            }
          }
        }
      }
    }
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.