Scaling ELK stack for monitoring performance of my servers

Elastic Stack Elasticsearch
1

Hi, I am new to ELK. I have to monitor daily performance of different servers of mine on my Kibana Dashboards (monitoring basically CPU, disk usage, health, traffic etc).

Here I am having 5 different servers. I have my ELK setup on my separate EC2 instance where all real time visualizations, etc will take place (My EC2 instance have configurations of t2.small). Data will be sent from those servers with help of beats (starting metricbeat on each server) coming to this EC2 instance. As here it may be needed to scale my elasticsearch using multiple nodes as data will be growing gradually each day coming from those servers, I wish to use separate nodes to store data (indexes) for each different server.

But I am facing issues in the SETUP.

I am confused how to add extra nodes, how to load balance, how to assign replicas, how to allocate master nodes, data nodes etc. so that my data gets properly indexed and accessible in the long run for performance comparison when required. I need expert advice here if my use case is clear to you.

Thanks in advance.

2

Do you really need to scale?

You have a N-tier application here:

  • logstash
  • elasticsearch
  • Kibana

To scale Logstash, you need a queue system in front, to load balance data across nodes.

To scale elasticsearch, the best is to read tutorial.

Logstash and Kibana will speak to your ES cluster as a single node, this is seamless.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.