Prod Architecture design document

[shroh (Mulligan)]

Hi Experts,

We are considering onboarding ELK on production with multi node architecture.

3 ES data nodes, 3 ES master nodes, 2 Logstash, 2 Kibana.

Main challenges:

1. We will be using AWS for the infra and challenge is to replicate it to other regions.
2. Where do we store the backups of the EBS volumes that we are gonna use.
3. What are the high availability considerations?

If anybody has a reference architecture on AWS, can you share that?

[warkolm (Mark Walkom)]

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats feels left out

I'd use snapshots to S3 instead of volume level ones, there is no way to quiesce Elasticsearch and it's unlikely to play nicely with your proposed model.

If you are running in a region with multiple zones, put nodes in each and then make them aware, eg https://www.elastic.co/guide/en/elasticsearch/reference/5.6/allocation-awareness.html

[shroh (Mulligan)]

Thanks Mark, we will be using EBS attached to the AWS EC2 as the primary storage for the instances, and S3 will be used for backup storage for snapshots.

High availability is needed for two main reasons:

1. Doing rehydrations as part of AWS strategy every 30 days, The DR stack should be during that time.
2. In case on region fails, we need a dr site.

Obviously in one region the instances and EBS volumes will be in different availability zones.

[shroh (Mulligan)]

All being said, do you have any reference architecture that we can use as far as AWS is considered?

[mujtabahussain]

Make sure to turn of ARP caching on nodes in AWS. That thing bit in my proverbial behind.