We are looking into using ELK to log 35 TB of business events generated. The data can even go bigger and come to sizes around 70-80 TB in the future.
The use-case is to allow real-time searches through Kibana to see failing/successful interactions per event type.
Because the data is huge, I thought, there can be 2 indexes: First index for the last 1 month of data, Second for the whole data (70 TB)
From the second index, will it be possible to use Kibana to create dashboards?
Another approach is to keep ELK for the first index, and use Hadoop environment for the whole data to create batch reports.