Change default replica factor for monitor* index?

Elastic Stack Elasticsearch
1

I want to increase the number_of_replicas for all the .monitoring* index created by x-pack. I realize they are using templates such as .monitoring-es-2.

Do I just create another index /_template/.monitoring-es-3 and that's it? where do I tell x-pack to create index from the new template not the old one? should I delete /_template/.monitoring-es-2 when I created 3?

PUT /_template/.monitoring-es-3
{
  "order": 0,
  "template": ".monitoring-es-3*",
  "settings": {
    "number_of_replicas": 4
  }
  ... # all other settings and stuff
}

Does this look right? Thank you!

2

How come you want so many replicas for this?

3

I just want to increase the reliability of internal indexes. If anything goes wrong, internal indexes should be the last one to cause problem.

For example, let's say I have a data index that has 3 replica, so I should be able to afford losing 3 nodes in the same time. However that's not true because the replica for monitoring indexes are 1, so when 2 nodes go down, we lost shards for monitoring indexes, which causes elasticsearch cluster to be red status, which means we cannot read or write to the whole cluster. Even though our data index has 3 replica, it doesn't help.

I hope this makes sense. Thank you!

4

Oh 4 in the post is just an example, I want it to be greater than 1.

5

Ok, so you need to balance the risk of losing 3 nodes with the cost of extra shards. One of the biggest problems we see with clusters is over-sharding and the resource cost of that.

So what is the risk of losing 3 nodes?

6

Also, you should really store Monitoring data in a separate cluster, it's the best way to ensure there is no impact and also means that you can get access to data if your main cluster is unavailable.

7

The monitor index is relatively small comparing to our data. So we're perfectly fine to increase the replica of the monitoring shard. Increasing replica for that index from 1 to 5 will not impact the cluster size we choose.

For current situation, we're ok to increase the replica for monitoring index.

8

Maybe have a look at https://www.elastic.co/guide/en/elasticsearch/reference/6.x/index-modules.html#dynamic-index-settings - index.auto_expand_replicas.

9

Interesting. It's very useful actually. But looks like this is dynamic settings so I have to update this setting after the index get created, I was hoping to avoid this manual part. That is why I was looking at the monitoring template.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.