Added a data node to an existing cluster, node joins, but data doesn't relocate to it

Elastic Stack Elasticsearch
1

Hi,

I have a support case open for this, but thought I'd post something here while waiting to hear back. We spun up a new data node in an existing cluster. The rest of the facts:

  1. The node joins the cluster
  2. The master reports (on the cluster pulse marvel dashboard) that it is relocating shards to it from one of the existing nodes.
  3. No shards move to the new node. There is no information about why the shards aren't relocating on either the data node where the shards are trying to move from, the currently elected master, or the new data node.
  4. I can share a folder on the data volume of the new node and write a file to that share from the data node where shards are trying to relocate from.
  5. There is enough disk space on the volume (3 TB free)
  6. elastic did create the F:\Elasticsearch\data\clustername\nodes\0 directory on the data volume, and in that directory is a 0 KB node.lock file.

The .yml is the same as on the other nodes in the cluster (except for the node name).
The version of ES and java are the same.
OS is the same

How can I get more detail about why data isn't actually relocating?

Thanks,
Casie

2

We found the issue. The license plugin was installed on the other nodes in the cluster but not this one. So, it was joining the cluster, but then disconnecting. We couldn't see it disconnecting until we enabled debug logging. After we installed the license plugin and restarted elasticsearch, shard began relocating to the node almost immediately.

The watcher plugin was also installed on the other nodes and not the new data node. We installed it on the new data node also, but we don't think the watcher plugin not being there was related in anyway to the data not relocating.