# Can I avoid elasticsearch monitoring index from moving to my temporary nodes?

**URL:** <https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128>\
**Category:** Elasticsearch\
**Created:** [February 13, 2024, 6:53am UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128 "2024-02-13T06:53:46Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Churchill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/churchill/32/131677_2.png) [@Churchill](https://discuss.elastic.co/u/Churchill)\
**Post date:** [February 13, 2024, 6:53am UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/1 "2024-02-13T06:53:46Z")

</div>

Good day,

I'm currently maintaining a cluster with 4 permanent nodes and 3 temporary nodes. How it works is, during work hours, our temporary nodes will be started automatically, and will be shutdown after work hours. The problem is that the .monitoring-es-7\* indices are being moved to the temporary nodes every time it starts. During the shutdown process of the temporary nodes, I exclude the temporary nodes from the cluster, and the .monitoring indices becomes UNASSIGNED shards and doesn't want to move to the permanent nodes. This leaves us a red state on our cluster. I have to manually delete the .monitoring index to bring it back to green state. Eventually, the monitoring index will be generated again and will be placed in our permanent nodes, but, it gets transferred back to the temporary nodes when the temporary nodes starts the next day.

Is there a way to tell Elasticsearch not to place the monitoring indices to our temporary nodes?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 13, 2024, 8:34am UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/2 "2024-02-13T08:34:29Z")

</div>

> [@Churchill](#):
>
> and the .monitoring indices becomes UNASSIGNED shards and doesn't want to move to the permanent nodes

Why is this exactly? I.e. what does the [allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) say about these shards? Excluding nodes from the cluster with an allocation filter won't itself cause any shards to become `UNASSIGNED` so I think something else is going on here.

---

<div class="post-metadata">

**Author:** ![Churchill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/churchill/32/131677_2.png) [@Churchill](https://discuss.elastic.co/u/Churchill)\
**Post date:** [February 13, 2024, 10:17am UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/3 "2024-02-13T10:17:06Z")

</div>

Hi @DavidTurner! I'm not sure if this plays part of the issue, but upon creation of our temporary nodes, we're setting cluster.routing.rebalance.enable to replicas to ensure that only replicas will be relocated to the temporary nodes. Then we set cluster.routing.rebalance.enable back to all after the temporary nodes gets destroyed. Here's the elasticsearch.yml of one of our temp nodes.

```auto
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
http.port: 9200
search.max_buckets: 65000
cluster.routing.allocation.enable: all
cluster.routing.rebalance.enable: none
cluster.routing.allocation.allow_rebalance: always
cluster.routing.allocation.node_concurrent_recoveries: 200
xpack.security.enabled: false
xpack.monitoring.enabled: true
cluster.name: platform-elasticsearch

node.name: temp-node-01
network.host: ["x.x.x.x","127.0.0.1"]
discovery.seed_hosts: ["x.x.x.x","x.x.x.x","x.x.x.x"]
cluster.initial_master_nodes: ["master-node-01", "master-node-02", "master-node-03"]
node.roles: [data]
network.publish_host: x.x.x.x

```

Here's the result when I check the settings of the .monitoring index.

```auto
curl -XGET "localhost:9200/.monitoring-es-7*/_settings?pretty"
{
  ".monitoring-es-7-2024.02.09" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".monitoring-es-7-2024.02.09",
        "format" : "7",
        "creation_date" : "1707486559155",
        "number_of_replicas" : "1",
        "uuid" : "2hYLQ2nGR4e7rM16xX6dLA",
        "version" : {
          "created" : "7160299"
        }
      }
    }
  },
  ".monitoring-es-7-2024.02.07" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".monitoring-es-7-2024.02.07",
        "format" : "7",
        "creation_date" : "1707264000054",
        "number_of_replicas" : "1",
        "uuid" : "_HUw6V3mS9K1Goh7rCzWIw",
        "version" : {
          "created" : "7160299"
        }
      }
    }
  },
  ".monitoring-es-7-2024.02.08" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".monitoring-es-7-2024.02.08",
        "format" : "7",
        "creation_date" : "1707401542253",
        "number_of_replicas" : "1",
        "uuid" : "qidNlIgPQL6C2_ZlqjvMkg",
        "version" : {
          "created" : "7160299"
        }
      }
    }
  },
  ".monitoring-es-7-2024.02.11" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".monitoring-es-7-2024.02.11",
        "format" : "7",
        "creation_date" : "1707609601292",
        "number_of_replicas" : "1",
        "uuid" : "xf2RDXH8QMii1l28nGll-A",
        "version" : {
          "created" : "7160299"
        }
      }
    }
  },
  ".monitoring-es-7-2024.02.12" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".monitoring-es-7-2024.02.12",
        "format" : "7",
        "creation_date" : "1707749582549",
        "number_of_replicas" : "1",
        "uuid" : "L4m1IlhkTOegxwdVzv0KBA",
        "version" : {
          "created" : "7160299"
        }
      }
    }
  },
  ".monitoring-es-7-2024.02.13" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".monitoring-es-7-2024.02.13",
        "format" : "7",
        "creation_date" : "1707782400198",
        "number_of_replicas" : "1",
        "uuid" : "2TZnmirvTd6Stwu9PuqYDw",
        "version" : {
          "created" : "7160299"
        }
      }
    }
  },
  ".monitoring-es-7-2024.02.10" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".monitoring-es-7-2024.02.10",
        "format" : "7",
        "creation_date" : "1707523200335",
        "number_of_replicas" : "1",
        "uuid" : "8lCqa4obTs20U7tZzsOWqA",
        "version" : {
          "created" : "7160299"
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 13, 2024, 10:37am UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/4 "2024-02-13T10:37:00Z")

</div>

> [@Churchill](#):
>
> I'm not sure if this plays part of the issue

Nor am I, but allocation explain will clarify.

---

<div class="post-metadata">

**Author:** ![Churchill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/churchill/32/131677_2.png) [@Churchill](https://discuss.elastic.co/u/Churchill)\
**Post date:** [February 13, 2024, 11:05am UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/5 "2024-02-13T11:05:08Z")

</div>

This is what I get from the last allocation explain i did:

```auto
node_left
cannot allocate because all found copies of the shard are either stale or corrupt.

```

I'm not sure why it would become stale or corrupt when I just excluded the temp node from the cluster.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 13, 2024, 11:07am UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/6 "2024-02-13T11:07:49Z")

</div>

> [@Churchill](#):
>
> I'm not sure why it would become stale or corrupt

Nor am I, but the _full_ allocation explain output will clarify.

---

<div class="post-metadata">

**Author:** ![Churchill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/churchill/32/131677_2.png) [@Churchill](https://discuss.elastic.co/u/Churchill)\
**Post date:** [February 13, 2024, 11:34am UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/7 "2024-02-13T11:34:37Z")

</div>

As a work around, I just added the following to the shutdown script to ensure that all .monitoring indices gets transferred to our master nodes.

```auto
curl -X PUT "localhost:9200/.monitoring-es-7*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._name": "master-es-nodes-*"
}
'

```

```auto
curl -X PUT "localhost:9200/.monitoring-kibana-7*/_settings?pretty" -H 'Content-Type: application/json' -d'
{
  "index.routing.allocation.include._name": "master-es-nodes-*"
}
'

```

I tested and it did moved all .monitoring indices to the master nodes. I guess I'm good for now. Thanks! 🙂

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 13, 2024, 12:15pm UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/8 "2024-02-13T12:15:00Z")

</div>

Hmm that's what I thought you meant when you said you were "excluding the temp node from the cluster". What did you mean if not adjusting the allocation filters?

Also you're setting index-level filters in those API calls, that's a lot less efficient than [cluster-level shard allocation filtering](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-shard-allocation-filtering).

---

<div class="post-metadata">

**Author:** ![Churchill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/churchill/32/131677_2.png) [@Churchill](https://discuss.elastic.co/u/Churchill)\
**Post date:** [February 13, 2024, 1:11pm UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/9 "2024-02-13T13:11:29Z")

</div>

> [@DavidTurner](#):
>
> Hmm that's what I thought you meant when you said you were "excluding the temp node from the cluster". What did you mean if not adjusting the allocation filters?

The way we exclude the temp nodes from the cluster is via:  
cluster.routing.allocation.exclude.\_ip: "x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x".. for some reason this makes the monitoring indices go UNASSIGNED and will not move to other nodes.

> [@DavidTurner](#):
>
> Also you're setting index-level filters in those API calls, that's a lot less efficient than [cluster-level shard allocation filtering](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-shard-allocation-filtering).

Thanks for this! I'll use the cluster-level filtering instead.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 13, 2024, 1:31pm UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/10 "2024-02-13T13:31:54Z")

</div>

> [@Churchill](#):
>
> The way we exclude the temp nodes from the cluster is via:  
> cluster.routing.allocation.exclude.\_ip: "x.x.x.x, x.x.x.x, x.x.x.x, x.x.x.x".. for some reason this makes the monitoring indices go UNASSIGNED and will not move to other nodes.

Ok, setting `cluster.routing.allocation.exclude._ip` itself cannot cause any shards to become UNASSIGNED. Unless you've found a weird bug ofc. We'd really like to see the full allocation explain from that situation to understand it.

---

<div class="post-metadata">

**Author:** ![Churchill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/churchill/32/131677_2.png) [@Churchill](https://discuss.elastic.co/u/Churchill)\
**Post date:** [February 13, 2024, 3:47pm UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/11 "2024-02-13T15:47:15Z")

</div>

> [@DavidTurner](#):
>
> Ok, setting `cluster.routing.allocation.exclude._ip` itself cannot cause any shards to become UNASSIGNED. Unless you've found a weird bug ofc. We'd really like to see the full allocation explain from that situation to understand it.

I'll provide the full allocation explain when I encounter the issue again. Thank you! 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 12, 2024, 3:47pm UTC](https://discuss.elastic.co/t/can-i-avoid-elasticsearch-monitoring-index-from-moving-to-my-temporary-nodes/353128/12 "2024-03-12T15:47:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.