Hi,
I'm building an ES cluster for application logs. The problem is that we have a lot of apps and a lot of environments. Every app and every env can have different retention times, so I can have let say over 100 apps that are logging into ES.
So far I tried to create daily basis indices for every app and every env. Which gives me precise control of retention times. But I run out of maximum indices capacity very quickly.
Next attempt was to use Rollover API and do not create daily indices. Which is more maintainable and indices are not created so often, but still. I have a feeling that I will have the issue in the future.
All apps are using ECS for structuring their logs, so I can group them into shared indices, there will be no mapping conflict. But I will lose retention control. It can happen that one app will be available for a month and the second for a few days.
I'm aware that I will need to make some compromise between precise retention control and maintainability of ES cluster.
Do you have any recommendations or best practices on how to structure the indices, templates, and ILM actions? I have terraform for hard work so I'm not afraid of granular configuration.
Thanks