Setting up Multi-node Architecture of ELK for log monitoring

Elastic Stack Elasticsearch
1

Hi all,
I have been taking care of ELK project for the last 2 years in my organization and had been running elasticsearch( version 6.x.x) and kibana as a part of our monitoring system.
Recently we have to planned to scale up our system and move to the multi-node setup of elk.
Following are the details of my Cluster:

  1. 1 Co-ordinating node.
  2. 3 master node
  3. 2 or 3 data node.

We aim to keep 90 days of data in our cluster. Thus as per our requirement, we will be keeping 30 days on 1st data node and rest 60 data on another data node.

Our 1 day log size ranges between 150-200 GB, which is pushed into elasticsearch in real-time.

My questions are-

  1. Should I keep Larger no. of CPU and High RAM on the master node. ( Higher means 10+ CPU and Higher Ram means 20+ GB)?

  2. For Data node, I have 2 options-
    a) Use 3 machines of 10TB SSD, 24 CPU, and 64GB RAM.
    b) Use 2 machines, one of 10TB SSD 24CPU and 32 GB RAM another of 20 TB SSD,
    24 CPU and 64 GB RAM.

  3. Lastly, Whether I should use individual physical machines of each node or I should go for virtual machines for data node?

Note: We aren't going of replication of any data. Each property will log its data into individual indices. Thus daily we will be creating 60 separate indices.

Please Revert asap.

Thanks in advance.

2

Why so many indices created every day? Given the data volume it sounds like you will end up with a lot of small shards even if you do not use a replica, which is inefficient and can cause performance problems.

I would recommend using fewer larger indices. If this is not an option I would recommend going with weekly indices rather than daily and make sure each index has a single primary shard.

Dedicated master nodes typically do not need a lot of resources and can be quite small. For data nodes for the data nodes I think option a) sounds better as storing karge data volumes often is limited by the amount of heap.

3

Hi Chris,
Thanks for the quick reply.

I would recommend using fewer larger indices. If this is not an option I would recommend going with weekly indices rather than daily and make sure each index has a single primary shard.
--- Actually, Currently, we take the daily backup of indices on S3 once they exceed the age of 30 days i.e. on the 31st day all the indices of 1 day will be moved to S3.

On requirement, we bring back that indices from the s3 and move it back to our cluster. Hence, playing on indices created on a daily basis seems convenient to us.
Is anything better we can do to enhance our performance given this condition??

Also, to keep the no. of shards in control, we explicitly define the no. of shards for each index which ranges from 6 shards( or larger indices- size 15 -16 GB) to 1 shard ( index size- less than 100 MB).

What best practices we can follow in our current case??

4

15 GB is not very large so there is no need to have more than 1 primary shard for any of the indices. Have a look at this blog post for some practical guidelines. Another way to reduce the index count would be to consolidate indices as 60 indices per day sounds excessive. You could also go with weekly or monthly indices for smaller ones and daily for the few ones that are larger.

5

Thanks Chris...
Any more suggestion or best practices to keep in mind before deploying multi-node architecture on production??

6

This webinar may also be of interest.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.