So I've a really specific infrastructure where I need to store my "Older than 30 days" indices on COLD/WARM nodes. Those nodes have a S3 bucket (1 bucket for all 4 nodes) mounted as a filesystem on each node in /data/ folder. Of course, /data/ is set as path for those nodes to store indices etc.
Setup is : 4 Hot, 4 Cold/Warm, 15GB RAM each (7GB Heap)
What I'd like to ask is: When we are talking about 100GB of data daily (right now) and something like 500GB of data daily in the future - does an infrastructure like this make any sense?
We were testing this for a while now but some problems with stability occured, like, whole Elastic was exploding. It seemed like S3 + S3FS is too slow to work on such amounts of data. All HOT and COLD/WARM nodes have 15GB of RAM and a heap of 7GB - that is a setup for 100GB of data per day, we will of course expand it but the most important question is:
Does mounting S3 with S3FS as a filesystem for Elasticsearch indices in RHEL 7 make any sense or should I look for some other ways to store old data?
I know this is a very abstract question so really I will be very gratefull for any answers!