we would like to deal with one year of various source of logs. that range from 80^6 events per day to 1000 per day with kibana on top to have reporting and dashbord of activity from those different sources.
we plan to have different sources of input so at the end we will have to deal with a collection of 365*5 index
all index collection will be standardized as much as possible in terms of idexed fields
as a POC we tried to index 6 month of logs from only one source and hited "memory heap" and "too many files open" and encountered some latency in the kibana search.
as far as I understand ES + kibana is most used for short time analysis not realy for long lasting log analysis.
does ES is suitable for this kind of task and will it support thhis kind of scaling
and what will be the best architecture we can eploy to cover this kind of task.