Cusotm ILM policy

Hello, we are currently employing the Elastic agent on Kubernetes, managed by Fleet, and have incorporated various Elastic integrations such as Kubernetes, Auditd, Audit_manager, and System. Our primary goal is to monitor both the logs and metrics originating from our server and Kubernetes cluster.

For testing purposes, we have established a straightforward ILM (Index Lifecycle Management) policy, which initiates index rollover after 10 minutes and proceeds to delete indices after 20 minutes.

resource "elasticstack_elasticsearch_index_lifecycle" "im_ilm" {
  name = "ilm-policy-${var.cluster.name}"

  hot {
    set_priority {
      priority = 0
    }
    rollover {
      max_age = "10m"
    }
  }
  delete {
    min_age = "20m"
  }
}

Additionally, we have crafted a custom component template and assigned the ILM policy to it.

resource "elasticstack_elasticsearch_component_template" "custom_template_playground" {
  name = "metrics-${var.cluster.name}@custom"

  template {
    alias {
      name = "metrics-playground@custom"
    }

    settings = jsonencode({
      index = {
        lifecycle = {
          name           = elasticstack_elasticsearch_index_lifecycle.im_ilm.name,
          rollover_alias = elasticstack_elasticsearch_index_lifecycle.im_ilm.name
        },
        number_of_replicas = "0"
        number_of_shards   = "1"
      }
    })
  }
}

Lastly, an index template has been created for all indices following a pattern similar to "metrics-*-playground6" as we aim to aggregate metrics specifically from the playground6 environment.

resource "elasticstack_elasticsearch_index_template" "custom_index_template" {
  name = "metrics-${var.cluster.name}"

  priority = 250
  index_patterns = ["metrics-*-${var.cluster.name}-*"]
  composed_of = [
    elasticstack_elasticsearch_component_template.custom_template_playground.name,
    "metrics-mappings",
    "data-streams-mappings",
    "metrics-settings",
  ]

  data_stream {
}


  template {

    settings = jsonencode({
      index = {
        number_of_replicas = "0"
        number_of_shards   = "1"
      }
    })
  }
}

It's worth noting that there are currently no linked index templates or attached linked indices in our ILM policy.

As evident from the default metrics ILM policy, there are 54 linked index templates and 69 linked indices. How can we automatically incorporate these indices and index templates into the custom ILM?

Did you create as explained in the documentation?

You need to create a custom template per dataset and clone the main template with a higher priority.

From what you shared your custom template does not follow the pattern from the documentation.

@leandrojmp Thank you so much for the prompt response. I manually recreated it through the Kibana UI, as explained earlier. However, the ILM policy is still not functioning as intended—rollover after 10 minutes and deletion of the index after 20 minutes.

As you can observe, three indices are associated with it.

The correct ILM policy is attached to the dataset, but it is not rolling over as expected through ILM. I performed this manually using the Dev Tool. Additionally, it is not deleting the old indices after 20 min.

Regarding this problem

You need to create a custom template per dataset and clone the main template with a higher priority.

It's difficult and unmanageable to create a custom template for each dataset. We are using auditd, auditmanager, kubernetes, system, elastic defend, agent, and fleet integration each one has many datastreams and datasets. Can't we make index template and custom template generic for metrics and logs for all playgorunds, staging and production? We are using namespace in the package policy so data is already segregated.

If you followed exactly what is explained in the documentation and create a template per dataset, than it should've work, not sure why it didn't work.

Can you get an ILM Explain on the index and share the response?

GET index-name/_ilm/explain

You can't, as mentioned currently every customization, be it custom life cycle policies, custom mappings or custom ingest pipelines, needs to be done on the dataset level, if you have for example an integration that has 10 datasets, you will need 10 custom templates.

Also, some customizations, will also need to be done using the namespace name, so if you have multiple namespaces you may need to have one template per dataset per namespace.

I faced the same issue last year when I started to use Elastic Agent and opened this github issue, things may improve in the future, but currently every customization is still on the dataset level.

Thank you once again. Here is the response from the index template and data stream.

{
  "index_templates": [
    {
      "name": "metrics-system.network-playground6",
      "index_template": {
        "index_patterns": [
          "metrics-system.network-playground6*"
        ],
        "template": {
          "settings": {
            "index": {
              "mode": "time_series"
            }
          },
          "mappings": {
            "_meta": {
              "package": {
                "name": "system"
              },
              "managed_by": "fleet",
              "managed": true
            }
          }
        },
        "composed_of": [
          "metrics-system.network@package",
          "metrics-system.network@custom",
          "metrics-system.network-playground6@custom",
          ".fleet_globals-1",
          ".fleet_agent_id_verification-1"
        ],
        "priority": 250,
        "_meta": {
          "package": {
            "name": "system"
          },
          "managed_by": "fleet",
          "managed": true
        },
        "data_stream": {
          "hidden": false,
          "allow_custom_routing": false
        }
      }
    }
  ]
}
{
  "data_streams": [
    {
      "name": "metrics-system.network-playground6",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-metrics-system.network-playground6-2023.11.18-000001",
          "index_uuid": "twL-xEogRTuS_VsBaI-Gow",
          "prefer_ilm": true,
          "ilm_policy": "metrics",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-metrics-system.network-playground6-2023.11.18-000002",
          "index_uuid": "ylf_V-NFRWaA9qjr--gFXg",
          "prefer_ilm": true,
          "ilm_policy": "metrics",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-metrics-system.network-playground6-2023.11.20-000003",
          "index_uuid": "XT7JqOG1TfSllD5lyjAiKw",
          "prefer_ilm": true,
          "ilm_policy": "test-ilm-playground6",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 3,
      "_meta": {
        "managed": true,
        "description": "default metrics template installed by x-pack"
      },
      "status": "YELLOW",
      "template": "metrics-system.network-playground6",
      "ilm_policy": "test-ilm-playground6",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-18T17:25:34.000Z",
            "end": "2023-11-21T00:49:17.000Z"
          }
        ]
      }
    }
  ]
}