Keep ILM policy on upgrade

Elastic Observability APM
,
1

Good day!
So we recently updated our hosted Elastic Cloud deployment to 7.4.1 to resolve the issue discussed in 503 Queue is full. The fix worked so all happy there. However, I came in this morning to find the disk nearly full...how could this be?

Well. When APM is upgraded, it looks like it resets the index templates and hence reset back to the default ILM policies. In our case we've created our own policies to be much tighter on retention, etc. so this explains why I now have a single, large 7.4.1 index for transactions!

I've combed though the docs and it looks like this can be overridden but it talks about pointing to a fields.yml file. Thing is

a) I only want to override the template for the -transaction, -span, etc templates
b) How does one have this fields.yml file be referenced when using hosted Elastic Cloud?

Or am I just misunderstanding how the template gets overridden?

Cheers

Dave

2

Darn, I thought I had it! I found the commented default config file

and constructed some YAML like:

apm-server:
  ilm:
    mapping:
      - event-type: "error"
        policy_name: "apm-REWIND"
      - event-type: "span"
        policy_name: "apm-REWIND"    
      - event-type: "transaction"
        policy_name: "apm-REWIND"
      - event-type: "metric"
        policy_name: "apm-REWIND" 

    policies:
      - name: apm-REWIND
        policy:
          phases:
            hot:
              actions:
                rollover:
                  max_size: "50gb"
                  max_age: "1d"
                set_priority:
                  priority: 100
            delete:
              min_age: "2d"
              actions:
                delete: ""

Which I then added under User Setting Overrides. It took it..but then wiped it :frowning: I'm guessing then that this isn't possible to change in the hosted Elastic Cloud APM?

3

Hi @dnorth98,
sorry to hear you are running into issues with ILM. We plan to release support for editing policies directly in the APM Server config file, which allows to configure changes once and have them automatically re-applied on every version upgrade.
The solution you posted is not released yet. Therefore you need to manually repeat your policy changes on upgrades, e.g via the Kibana UI. Policy changes are applied to all indices attached to the policy, but the settings for the current phase are cached and will only take effect on entering the next phase. You could also remove and newly add policies to indices if that fits your use case better.

1 Like
4

Thanks for the reply! Yeah, I found what I had to do was:

  • Update the index templates to use my policies again
  • Update the existing indicies to remove ILM
  • Re-add ILM to each index

It was super time consuming with all the rollover. I think I'll be more careful about which upgrades I apply in future. Hosted does not mean "no work" :slight_smile:

Thanks again!

Dave

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.