Scaling ELK stack


(Manthan Patel) #1

Hi,

I have ELK stack on AWS. It has capacity of around 5000 UDP msg/sec.

I want to scale it up to capacity of 20,000 msg/sec. What would be the recommended architecture for ELK stack.

Will it help If I keep N number of small instance installed logstash on all instance behind load balancer and keep Elasticsarch cluster on all instances or I keep bigger single instance.

And what option is better for future upgrade. ?


(Mark Walkom) #2

FYI we’ve renamed ELK to the Elastic Stack, otherwise Beats feels left out :wink:

If you are running Elasticsearch and Logstash together, splitting them out is definitely the first step. Using a load balancer also helps.


(Manthan Patel) #3

Thank you :),
Will definitely keep Logstash and Elasticsearch on separate machines .

What do you suggest about architecture. I currently run on 16 GB of memory, 4 core with capacity of around 4500 to 5000 msg/sec.

Do we just upgrade machine or we go for multiple smaller instances ?


(Mark Walkom) #4

You shouldn't need that much memory for Logstash, but it is CPU heavy. Having a few smaller ones is likely going to be more effective.


(Manthan Patel) #5

Thank you for help :slight_smile:


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.