Node with zero shards fails to start with error "Node is started with node.data=false, but has shard data"

[tomhe]

I'm trying to remove the data role from one node by first excluding the node from shard allocation with:

PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._name" : "node-001"
  }
}

When the node showed 0 shards I changed "node.data" to false and restarted it, but the node failed to start with this error:

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Node is started with node.data=false, but has shard data: [/usr/share/elasticsearch/data/nodes/0/indices/KdyBcBhwQhmrcvW9mPy5eQ/4, /usr/share/elasticsearch/data/nodes/0/indices/KdyBcBhwQhmrcvW9mPy5eQ/7, /usr/share/elasticsearch/data/nodes/0/indices/KdyBcBhwQhmrcvW9mPy5eQ/2, /usr/share/elasticsearch/data/nodes/0/indices/KdyBcBhwQhmrcvW9mPy5eQ/9, /usr/share/elasticsearch/data/nodes/0/indices/KdyBcBhwQhmrcvW9mPy5eQ/3, /usr/share/elasticsearch/data/nodes/0/indices/KdyBcBhwQhmrcvW9mPy5eQ/1, /usr/share/elasticsearch/data/nodes/0/indices/KdyBcBhwQhmrcvW9mPy5eQ/8, /usr/share/elasticsearch/data/nodes/0/indices/KdyBcBhwQhmrcvW9mPy5eQ/6, /usr/share/elasticsearch/data/nodes/0/indices/YJXzvwdRR1qU6I23pr0SXQ/4, /usr/share/elasticsearch/data/nodes/0/indices/YJXzvwdRR1qU6I23pr0SXQ/2, /usr/share/elasticsearch/data/nodes/0/indices/YJXzvwdRR1qU6I23pr0SXQ/0, /usr/share/elasticsearch/data/nodes/0/indices/YJXzvwdRR1qU6I23pr0SXQ/9, /usr/share/elasticsearch/data/nodes/0/indices/YJXzvwdRR1qU6I23pr0SXQ/3, /usr/share/elasticsearch/data/nodes/0/indices/YJXzvwdRR1qU6I23pr0SXQ/1, /usr/share/elasticsearch/data/nodes/0/indices/YJXzvwdRR1qU6I23pr0SXQ/5, /usr/share/elasticsearch/data/nodes/0/indices/YJXzvwdRR1qU6I23pr0SXQ/6, /usr/share/elasticsearch/data/nodes/0/indices/bagM1fomTDm_dwV3mymIpg/4, /usr/share/elasticsearch/data/nodes/0/indices/bagM1fomTDm_dwV3mymIpg/2, /usr/share/elasticsearch/data/nodes/0/indices/bagM1fomTDm_dwV3mymIpg/0, /usr/share/elasticsearch/data/nodes/0/indices/bagM1fomTDm_dwV3mymIpg/3]. Use 'elasticsearch-node repurpose' tool to clean up

There are no shards allocated to his node, so why am I getting this error?

[tomhe]

I found the cause: There were shards from three closed indices on that node.

[tomhe]

Ran into the same situation again on the next node that I no longer wanted to use as a data node. The node has zero shards allocated.

sh-4.2# bin/elasticsearch-node repurpose -v
------------------------------------------------------------------------

    WARNING: Elasticsearch MUST be stopped before running this tool.

Obtaining lock for node
Collecting shard data paths
Paths to clean up:
  /usr/share/elasticsearch/data/nodes/0/indices/myrWs3pnSmuZwTplN3XKdw/3
Indices affected:
  important_index_in_use
Found 1 shards in 1 indices to clean up
Node is being re-purposed as master and no-data. Clean-up of shard data will be performed.
Do you want to proceed?
Confirm [y/N]

(Index is not named "important_index_in_use", but I changed it here in the post.)

This index is in use. Why am I seeing shard data here even though the node has zero allocated shards?

[DavidTurner]

Elasticsearch cleans up unused shard data from a data node when the shard is fully allocated (i.e. green health) none of its shards are relocating. If that wasn't the case when this node was shut down then the data might not have been cleaned up. If this index reports green health then you can safely remove the shard data on this node.