How can I limit the data retention on our hosted plan? Is there an easy way to limit to, for example, 30 days of data?
Hi @jsumners_knock,
depending on the version you are running you have following options:
From 7.5 on you can configure ILM policies directly in the APM Server configuration, allowing you to add a delete phase for indices. If you restart APM Server with new config values, ensure to set apm-server.ilm.setup.overwrite: true so the new policies are overwriting existing ones. Such policy changes are going to be applied when indices enter the next phase.
For 7.x versions < 7.5 see the manual ILM setup guide.
Alternatively you can update ILM policies via Kibana UI.
You might also be interested in the general guide on how to reduce storage.
I'm not an ES administrator and have no desire to become one. I'd really prefer a simple UI that does all of this (one of the reasons to use the hosted product IMHO).
After performing an upgrade so that I can try to follow your instructions, I have come to:
It looks like the upgrade has included the exact policy I'm looking for on at least most of the new indices. Do I need to do anything for the one labeled "apm-7.5.1*"? Is it going to grow ad infinitum as it has done previously? Or is it limited by the ones that have the "apm-rollover-30-days" policy?
What can I do to clean up the old apm-7.3.2 stuff? Do I need it any longer? Has that data been migrated into the 7.5.1 data?
@simitt do you have any more information for these questions?
The screenshot you posted shows the created templates and index patterns. For checking that ILM policies are actually applied to your indices please check the Indices tab. You should see indices for all event types in the form of apm-7.5.1-error-000001. You can click on Lifecycle status and choose Managed, which filters for all indices managed with ILM - if the 7.5.1 indices show up there ILM is applied as expected. You can also click on an index and see which lifecycle policy is applied.
Regarding your question whether or not indices grow infinitly: the default policy in 7.5 rolls over after 30 days or when reaching a size of 50gb, whichever limit is reached first. This means that a new index is created at those limits. The default policy does not contain a delete phase though. So if you want to remove data, you need to add a delete phase. Please refer to my previous answer where I linked to according documentation.
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.