We are using elasticsearch cloud and enabled the APM component. Now, after a month or so, we have about 150 indices of APM, called "apm-7.5.1-metric/span/error-000001"
I understand that I need to configure ILM for those, but I can't understand how does the indices even switch between the lifecycle phases.
Can I limit the number of APM indices? Where can I find a generic ILM policy so I'll only have max of 10-20 indices of APM?
I recommend taking a read through the APM ILM documentation when you have a chance. There's (hopefully) a lot of useful information about how ILM works available there!
The default ILM policy for APM rolls data over (creates a new index) when the index reaches a maximum size of 50gb or a maximum age of 30 days. After 30 days, data will be moved to the warm phase. The default policy does not define a delete phase, so your total number of indices will continue to grow.
Is there a reason why you want to limit the number of APM indices to 10-20? You could accomplish this by increasing the maximum size of each index, which would lead to a smaller total number of indices, but 50gb is generally a good maximum size. If you're more so concerned about the amount of data you're storing, you can define cold and/or delete phases. A cold phase with frozen indices keeps data searchable without occupying heap, while a delete phase removes the index completely.
You can amend the default APM ILM policies in Kibana. Here's an example config that deletes error and span indices after 60 days, and freezes transactions and metrics after 90 days.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
