How to plan/define needed clusters

Elastic Stack Elasticsearch
1

Hello,

I'm working on setting up an elastic search environment within a kubernetes cluster (on MS Azure AKS), and am reading through the Elastic Cloud on Kubernetes [1.5] | Elastic documentation. It explains how to setup clusters and configure them.

The cluster will be used for gather logs of our various microservices running inside the kubernetes cluster (they are .NET services and will be sending their logs using the Serilog elastic search sink), and, more importantly, indexing various models of our application specific data for fast searching. Lastly, I also want to monitor the performance of all clusters we end up deploying.

From the Monitoring overview | Elasticsearch Guide [7.12] | Elastic docs, I gathered that monitoring of an elastic search cluster, should be done in a separate monitoring cluster. This got me thinking, what else, besides monitoring, constitutes the need for a separate cluster. A cluster can have multiple indices, so can everything else (logs & search indices) be put in one cluster, or is there some guidelines as to what should be put in a separate cluster?

I could not find this information anywhere. There's a lot to be found about node planning and shard sizing, but not about clusters in general.

2

It's a best practise, but it's not mandatory.

Usually you wouldn't put logs (text) and search data in the same cluster.

You might also separate prod and non-prod clusters.
You can also consider moving high velocity data into it's own cluster with different specs to others.
And you might also want to keep data geolocated to it's source, to save data transfer costs

The last three are optional.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.