Seek Recommended Storage Type for Data Retention

Elastic Stack Elasticsearch
1

Dear Team,

We would like to seek your recommendation on log data retention, as we are implementing the Elastic Stack on-premise. To align with regulatory requirements, we must store the data on NAS storage using the Hot, Warm, Cold, Delete, and Snapshot features.

Given this requirement, could you please advise which external storage solution is recommended for our RedHat Linux environment—NFS share, SMB, or iSCSI?

Thank you in advance for your response.

2

I believe the cold storage tier relies on searchable snapshots, which is a feature that require a commercial license. If you have this or plan to get one I would recommend you reach out to Elastic and see if a solution architect can help with sizing and capacity planning.

Without searchable snapshots you are otherwise looking at a hot-warm architecture. This does , as outlined in this old blog post, rely on having nodes with different hardware profiles in the different tiers. The hot tier usually requires fast local SSD storage as it can be very I/O intensive, but this will to some extent depend on the use case’s ingest volumes, query performance, and retention requirements which you have not at all provcided any detail on.

If all nodes are required to use exactly the same type of storage a hot-warm type architecture may not be at all suitable. As outlined in the documentation many types of networked storage are not suitable for use with Elasticsearch data nodes as they offer poor random read/write performance. When networked storage is benchmarked they often provide statistics for reading and writing of large sequential files, which is not how Elasticsearch typically use storage. I would also recommend reviewing the the recommendations on optimising for indexing speed.

If we disregard any performance implications of using networked storage it is important that any storage used with data nodes behave and provide exactly the same durability guarantees as local storage, so I would suspect iSCSI might be the best option.

3

I agree with all that Christian said but particularly this bit. Local storage is best and iSCSI is the closest of the three options available to you.