I am storing a lot of logs from an cloud application (architecture logs, application logs and syslogs) on Elasticsearch.
My idea was to keep -at least- logs of last 30 days on Elasticsearch to allow analysis on application performance by using Kibana dashboards. I am also using one index for all the data.
Some days ago, an engineer told me that if I keep such amount of data and use just one index, the performance will be por because Elasticsearch, if I have 1TB of data using one index, ill try to get 1TB of RAM to load such index. It sounds a Little bit weird to me then I come here to ask to experienced people.
Could somebody tell me what is the best practice? Should I Split my Elasticsearch indexes by rolling them by date or keep just one index for the whole month? Is it true that such only index will eat all the RAM?
Also, is it OK to store data on Elasticseach for historic analysis or shall I export it to a bigdata DB?
Thanks in advance