How to define delete phase of apm server policies

[Kay_Khan]

Hi friends

I would like to remove the delete phases of every apm ilm policy or at the very least set it to delete in 3 years. Right now we can't seem to get it working at all.

Aside: Can i apply the setting globally across all of APM's created lifecycle policies. If not is there some list of policies somwhere?

Here is my current attempt.

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-server-prod
  namespace: elastic-system
spec:
  version: 8.7.1
  count: 1
  elasticsearchRef:
    name: "elasticsearch-prod"
  kibanaRef:
    name: "kibana-prod"
  http:
    service:
      spec:
        type: NodePort
  podTemplate:
    spec:
      containers:
      - name: apm-server
  config:
    http.enabled: true
    http.port: 5068
    monitoring.enabled: false
    setup.template.settings:
      index:
        routing.allocation.require.type: "monitoring"
        anonymous:
          rate_limit:
            event_limit:  300
            ip_limit:  1000
          allow_service: ["admin-dashboard", "frontend"]
    apm-server.rum.enabled: true
    apm-server.rum.allow_origins: ["*"]
    apm-server:
      template.settings:
        index:
          routing.allocation.require.type: "monitoring"
      ilm:
        enabled: "auto"
        setup:
          enabled: true
          overwrite: true
          require_policy: true
          policies:
            - name: "traces-apm.rum_traces-default_policy"
              policy:
                phases:
                  hot:
                    actions:
                      rollover:
                        max_size: "50gb"
                        max_age: "30d"
                      set_priority:
                        priority: 100
                  delete:
                    min_age: "1095d"
                    actions:
                      delete: {}

GET _ilm/policy/traces-apm.rum_traces-default_policy
But you can see nothing has changed.

{
  "traces-apm.rum_traces-default_policy": {
    "version": 1,
    "modified_date": "2023-06-05T14:19:02.806Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_size": "50gb",
              "max_age": "30d"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      },
      "_meta": {
        "package": {
          "name": "apm"
        },
        "managed_by": "fleet",
        "managed": true
      }
    },
    "in_use_by": {
      "indices": [],
      "data_streams": [],
      "composable_templates": [
        "traces-apm.rum"
      ]
    }
  }
}

[simitt (Silvia Mitter)]

Hi @Kay_Khan
you can follow this guide for configuring custom ILM policies. You could configure one policy and then link it from multiple @custom component templates, if they are supposed have the same retention period.

The docs for default policies are currently not up-to-date. I created an issue for us to update the docs, you can take a look the details from the issue to get informed about the data streams.
You could alternatively also query all the @custom component templates from Elasticsearch (traces-apm*, logs-apm*, metrics-apm*).

[Kay_Khan]

Ok if i understand correctly the correct way to modify the policies for apm is to;

1. Create a new ILM policy with the desired configuration
2. Update the @custom component template so that it is using this new ILM policy
3. roll over the datastream.

How much of this can we define via yaml? and where should we define this

There was some documentation at one point where you could define apm-server.ilm (Configure Index lifecycle management (ILM) | APM User Guide [7.17] | Elastic) but this does not seem to work in 8.x

[simitt (Silvia Mitter)]

Yes these are the required steps. Configuring the ilm policies via kibana config is not supported. But you can leverage the ES API to update the component template with the custom settings.

There was some documentation at one point where you could define apm-server.ilm (Configure Index lifecycle management (ILM) | APM User Guide [7.17] | Elastic) but this does not seem to work in 8.x

Exactly, the apm asset setup (index templates, component templates, ILM policies, etc.) moved to Kibana from 8.0 onwards. Having multiple apm-servers with conflicting index setup configurations has caused problems in the past, therefore the setup is now centralized. This means it is not configurable anymore via apm-server.yml.

[Kay_Khan]

Okay thanks for confirming its no longer possible to codify these changes in our apm-server.yaml.

We have been following a standard when installing any elastic component and its a shame that we can't do this when it comes to the apm-server.

e.g filebeat and metricbeat look similar.

...
    setup.ilm:
      enabled: true
      policy_name: metricbeat-custom
      policy_file: /etc/indice-lifecycle.json
      overwrite: true

---

We understand and makes sense to centralise the creation of these resources, although we hope that you would take into consideration of allowing users to modify the @custom life cycle policy and for that to be the default ilm policy for all apm components via some .yaml configuration, so that we can codify the changes.