Hi,
- I am collecting logs from kubernetes cluster using filebeat, which is deployed as daemonset
- I ship logs using logstash to my elasticsearch cluster in one index
- I am using ILM for controlling the retention time for my kubernetes logs index.
This setup works fine so far.
Now I have following use case: I want to allow the dev teams to take control over the retention times for their logs.
One obvious solution would be to create one index for each namespace (or even app?) for example to be able to use dedicated lifecycle policies, which brings me into a position to always know in advance about each deployment taking place in order to be able to create templates, indices and lifecycle policies and become a potential blocker.
As far as I have learned elastic I refuse to believe that this is the only solution.
How do you guys deal with this conflict? On one hand you do not want to know about the deployments taking place in your kubernetes cluster, on the other hand you want to allow everyone to have control over their retention times (maybe replication too).
Appreciate any ideas
Thanks