Question for Logging Cluster - is 3master, 2data node is good?

Elastic Stack Elasticsearch
1

Hi everyone,

So we tried to move to ECK, and want to execute it perfectly or at least the right way...

My use case is for logging and APM,
Indexing around 7000ish/s
Search rate 50-100/s

Now we have an ECK cluster consisting of:

  • 3 Master
  • 2 Data
  • 1 APM

I'm planning to gradually grow the storage of data nodes until they can fully store our retention logs.

So here's the question

  1. Is it a good formation?
  2. Can it handle the traffic well?
  3. When there's a logging burst which node will have a hit, data or master node?
  4. I'm planning to stick with 2 data nodes (1 for replica) and keep resizing the storage, is it good or should I spawn multiple nodes and spread the shard?
  5. Should/can I change the master dedicated to master-data in this case?

Thanks!

2

Are these master only?
What are the specs?

3

yes, master only. All node have 4c/7gb usable RAM

4

There are a lot of unknowns to fully answer your question, but there are some general suggestions and pointers that can help you on your way using elastic and making your cluster robust.

  1. Is it a good formation?
    Of course it depends, but you are doing it right by having dedicated master only nodes. Having 2 data nodes is also a good thing so the cluster, like you say, can save replicas to a different host for fault tolerance. You'll probably notice the cluster puts some primaries on one node and other primaries on the other. That shouldn't matter and the cluster will take care of all those details. Having APM on its own server is also a good idea.
  2. Can it handle the traffic well?
    That's a big question with many dimensions but some that you'll want to think about are: ingest capacity, search responsiveness, data capacity.
  • Ingest capacity: You don't talk about how data is getting into your cluster but as your cluster grows it would be a good idea to add one or more dedicated ingest nodes. Your data nodes might be configured currently to be ingest nodes.
  • Search responsiveness: having your indices right sized and under an appropriate ILM (index lifecycle management) policy will go a long way to making sure your cluster is responsive. You don't say if you are using kibana, but it is often a good idea to put kibana on its own host and configure it to be a coordinating node as your cluster grows. You say you have 7gb of ram on the machines. That should be enough for master nodes, but you will notice an improvement generally as your data nodes get more ram up to 32GB.
  • Data capacity: Elastic scales well horizontally by adding more data nodes.
  1. When there's a logging burst which node will have a hit, data or master?
    Master only nodes won't be "hit" by the burst, but the data node will. Of course the master nodes will know about the data coming in but assuming everything is healthy they will handle it well.
  2. I'm planning to stick with 2 data nodes (1 for replica) and keep resizing the storage, is it good or should I spawn multiple nodes and spread the shard?
    Sounds like all the hosts are running in a VM. Nothing wrong with that but it is beyond the scope of what can be answered here. Usually as data ages with ILM shards get consolidated and become read only.
  3. Should/can I change the master dedicated to master-data in this case?
    Yes you can, but wouldn't recommend it.

In closing I like to approach an elastic cluster with the following in mind.
First priority is data integrity, that means having a cluster with the right sized indices, on a enough data nodes to have resiliency. Also, turn on backups if you haven't done so already, and link that to your ILMs.
Second priority is responsiveness. You can increase the number of replicas to improve performance if done correctly (example: 8 data nodes with 1 primary and 7 replicas, could be more performant than 1 primary and 1 replica, all other things being equal). Also, you should look into using ILM to make sure your data ages appropriately. Often slower response times are acceptable for older data. Depends upon your use case of course. There really is a third priority between the two being security, but not touching on that since it wasn't asked about.

Lastly, elasticsearch is a great product and platform that is very robust. The more you learn about it and manage it the more you'll be rewarded in what the system can do. Glad you are wanting to learn more and get it right from the start. We've all been there and you are in the right place. Finally please take these ideas as suggestions, and be open to experimenting and seeing what works best for your unique use case.

5

Hi! thanks for the really detailed and thorough answer... really appreciate it.

For now, we have this configuration:
3 ingest-data node - 1 primary 2 replicas 3 shard
3 dedicated master node
1 kibana node
1 apm-server node

ILM enabled with 30 days retention.

from your reply, I can assume that we're on the right track yes?
but, what are the master roles besides ensuring the cluster is healthy?

image
image1796×429 34.2 KB

it just sits there, like doing nothing

6

It sounds like you are on the right track to me. It's true that a healthy cluster with 3 master nodes won't look like it is doing much. By having at least 3 master nodes (with two of them master eligible) you are creating what the documentation refers to as "high availability" ([Node | Elasticsearch Guide [8.6] | Elastic](Node | Elasticsearch Guide [8.6] | Elastic). Could you run a cluster with master nodes and data nodes on the same host? Absolutely. As the cluster grows you'd probably want to switch to the master node only module you currently have. So if it is all the same you're saving yourself some work down the road.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.