Yes, these are a lot of completely different, independent applications.
Looks like the only solution is really to separate the applications to different indices, which brings quite a lot of overhead, which I by the way tried to address in my post to collect some experiences from other users, but somehow it looks like this is not a common issue: How you deal with ILM for kubernetes cluster applications?
I see following drawbacks:
- I need to create index templates and indices in advance before someone deploys an application, because I want to use ILM.
- My logstash pipeline will become pretty huge due a lot of "if else" statements to seperate application logs into different indices. This could become a problem like slower processing times and larger memory consumption, right? Also when something is wrong with the logs from one application the whole pipeline will fail and the shipping of all application logs will be impaired.
- Filebeat hint based autodiscover is sort of absolete in this case ain't it? It should provide to the developer the flexibility to control the configurartion of his logs without having to ask me to change the logstash configs. But due to the index separation I will become a blocker, since I am needed to apply the changes.
However. Can you maybe help me to get these things clear?