Question About Removing A Node From A Production ELK Stack

Elastic Stack Elasticsearch
1

Hello,

I have a question about removing a node from a production ELK Stack. We are removing a server from the stack to repurpose it for another use. After moving shards from the node that we are targeting for removal from the ELK Stack I see on that server in the data directory that there is still data. Is this just left over data that is safe to delete because it was "copied" to the other nodes in the stack?

Here is what I did:

I excluded the target node so that the shards would be moved to other nodes in the stack. In Kibana in DevTools I ran:

PUT _cluster/settings
{
  "transient" : {
    "cluster.routing.allocation.exclude._ip" : "%IP_OF_TARGET_SERVER_FOR_REMOVAL%"
  }
}

This ran over night and moved the data from the server to our other nodes. The next day I checked everything. Here are the commands I ran:

GET _cat/shards?v&pretty

No index is associated with the node that we are targeting for removal. Additionally when I ran:

GET _cat/allocation?v

and

GET _node/%%NAME_OF_TARGET_NODE%/stats/indices

All indication is that there is no data on the node that we are targeting for removal. But when I actually log into the node/server, I see in the directory that was the data directory that there is still a couple hundred gigs of data.

Is this still production data?

Is there another step I need to take to move our data before we actually remove this node from our ELK Stack?

Thank you for all your help!

2

It is somewhat surprising that there's still data on disk on the now-empty node, but if you shut the node down and the remaining cluster is in green health then whatever this data is it isn't important to Elasticsearch.

3

Thanks David, I'll do some testing to verify and report back here.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.