Which Storage type to be used for ELK on AWS?

Elastic Stack Elasticsearch
1

https://www.elastic.co/guide/en/elasticsearch/plugins/master/cloud-aws-best-practices.html

From the above it is recommended to use Instance storage, but then data is lost in the below scenarios:

  • The underlying disk drive fails
  • The instance stops
  • The instance terminates

What is the general practice, how can this be handled ?

2

Do you plan to run only one node in your cluster?

It's recommended to have 3 nodes (at least 2 data/master nodes + one master only node). In which case you'll have a copy of the data in the other nodes.

3

I have posted it as a questing in thread https://discuss.elastic.co/t/regular-elk-stack-on-aws/169997

4

Storage Type to be used is EBS

5

Up to you but this is not really what is recommended. Read https://www.elastic.co/guide/en/elasticsearch/plugins/6.6/cloud-aws-best-practices.html

6

Yes, you are right, but then with the EFS, there are performance issues.

7

and the next question what if I am using the EC2 Instance Storage ? Again if the instance dies, the data is gone

8

With Instance Storage, you're right, you'll need to make use of the features provided by Elasticsearch to make sure your data isn't lost.

First, have multiple nodes, preferably at least 3, and make sure to set index.number_of_replicas on each index to 1 or more. This ensures that Elasticsearch will keep copies of your data on multiple nodes - it will keep copies on index.number_of_replicas + 1 nodes. The higher you set that setting on each index, the more nodes that can die without losing data.

Second, use Snapshots to periodically take backups of your data to a storage service like S3. This will ensure that you can recover your data even if your entire cluster dies.

EBS does work, mostly, but be aware that Elasticsearch is sensitive to IO performance, and that EBS has IOPS limits not only per volume, but per EC2 instance - that is, for a given EC2 instance size, there's a limit to how many IOPS that instance can perform, no matter how many IOPS are provisioned for the attached volume, and going over your IOPS limits can cause very significant slowdowns in performance that are difficult to diagnose.

1 Like
9

Thanks Gorden !! Which EC2 Instance would be recommended ? M42xLarge ?

10

Which EC2 instance you should use depends almost entirely on the workload you'll have and your budget. I can't really give concrete guidance here other than in the most general terms:

A server that's running Elasticsearch and nothing else should typically top out at 64GB of RAM, a little less than 32GB given to Elasticsearch's heap and the rest left for the OS to use for filesystem caching, and if you need more performance beyond that, add more nodes rather than making existing nodes bigger.

Beyond that and the disk IO considerations I mentioned above, the most reliable way to determine your hardware needs is to benchmark. We have a custom tool for benchmarking Elasticsearch called Rally that can be customized pretty easily to replicate your workload.

11

Thanks Gordon, I am actually using three Master nodes across two availability zones. from a workload point of view, It's a brand new setup. I would take an example of a retail store like "amazon.com". I would be expecting a similar kind of load in about two years. So my EC2s should be able to survive for at least three years. Does these EC2 instances hold true for Data/Coordinating Nodes as well ?

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.