Node does not cluster setting [cluster.routing.allocation.include] filters

Elastic Stack Elasticsearch
1

Hello,

ES version 5.6.10
16 data nodes
3 master nodes

Can't perform any shard's relocation:

 Error while starting relocation
{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[graylog-es-master-68d84fd5ff-4kvl5][10.244.3.205:9300][cluster:admin/reroute]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "[move_allocation] can't move 8, from {graylog-es-data02-0}{h4tkkOIITZ68ywpS2fpG-A}{-Jqr5kAWSCmpJHcJlkfBAQ}{10.244.7.246}{10.244.7.246:9300}{ml.max_open_jobs=10, ml.enabled=true}, to {graylog-es-data05-3}{aDx6vKhZTbiLPVEnsA_Wpw}{pLouQGBRSE-OTm2Ph-4f7A}{10.244.9.89}{10.244.9.89:9300}{ml.max_open_jobs=10, ml.enabled=true}, since its not allowed, reason: [YES(shard has no previous failures)][YES(shard is primary and can be allocated)][YES(explicitly ignoring any disabling of allocation due to manual allocation commands via the reroute API)][YES(target node version [5.6.10] is the same or newer than source node version [5.6.10])][YES(no snapshots are currently running)][YES(ignored as shard is not being recovered from a snapshot)][NO(node does not cluster setting [cluster.routing.allocation.include] filters [_name:\"graylog-es-data01-0\"])][YES(the shard does not exist on the same node)][YES(enough disk for shard on node, free: [1.7tb], shard size: [19.2mb], free after allocating shard: [1.7tb])][YES(below shard recovery limit of outgoing: [0 < 2] incoming: [0 < 4])][YES(total shard limits are disabled: [index: -1, cluster: -1] <= 0)][YES(allocation awareness is not enabled, set cluster setting [cluster.routing.allocation.awareness.attributes] to enable it)]"
  },
  "status": 400

Any help will be appreciated.

2

What does your elasticsearch.yml look like?

Please upgrade, 5.X has been EOL for some time now :slight_smile:

3

It looks like you have some index/cluster settings that prevent the relocation.

4

At this time it is not possible to upgrade due to the use of software that only works with this version.

elasticsearch.yam 
cluster.name: "graylog-es"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 2
path.repo: ["/backup/clear_backup"]
and additional parameters passed via k8s ( elasticsearch deployed on k8s )
- name: node.master
          value: "false"
        - name: node.ingest
          value: "true"
        - name: node.data
          value: "true"
        - name: http.enabled
          value: "true"
        - name: xpack.security.enabled
          value: "false"
        - name: ES_JAVA_OPTS
          value: -Xms21g -Xmx21g
        - name: discovery.zen.ping.unicast.hosts
          value: graylog-es-discovery
        - name: discovery.zen.minimum_master_nodes
          value: "2"
        - name: index.store.type
          value: simplefs
        - name: thread_pool.bulk.queue_size
          value: "500"

I played with parameters as cluster.routing.allocation.exclude._name and cluster.routing.allocation.include._name, index.routing.allocation.exclude._name....
For sure something prevent the relocation, I just can not find what exactly

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.