[node-frozen] fatal exception while booting Elasticsearchjava.io.UncheckedIOException: java.io.IOException: Not enough free space [55991361536] for cache file of size [131852140544] in path [/usr/bin/elasticsearch-node-frozen/elasticsearch-8.15.1/data]

Elastic Stack Elasticsearch
1

I have a problem with adding a new frozen node.

At first, I have installed Elasticsearch of version 8.15.1 on my local machine by extracting tar.gz on Ubuntu 22.04. LTS. It is working fine. Then, I have added a cold node (also extracted from same .tar.gz). It is also working fine. Finally, I have tried to add a frozen node (also extracted from same .tar.gz) with

$ sudo -u elastic ./bin/elasticsearch --enrollment-token <enrollment_token>

but I have got an error during running at first time:

[2024-09-30T21:29:08,851][ERROR][o.e.b.Elasticsearch      ] [node-frozen] fatal exception while booting Elasticsearchjava.io.UncheckedIOException: java.io.IOException: Not enough free space [55991361536] for cache file of size [131852140544] in path [/usr/bin/elasticsearch-node-frozen/elasticsearch-8.15.1/data]
        at org.elasticsearch.blobcache@8.15.1/org.elasticsearch.blobcache.shared.SharedBlobCacheService.<init>(SharedBlobCacheService.java:378)
        at org.elasticsearch.blobcache@8.15.1/org.elasticsearch.blobcache.shared.SharedBlobCacheService.<init>(SharedBlobCacheService.java:338)
        at org.elasticsearch.searchablesnapshots@8.15.1/org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.createComponents(SearchableSnapshots.java:335)
        at org.elasticsearch.server@8.15.1/org.elasticsearch.node.NodeConstruction.lambda$construct$13(NodeConstruction.java:868)
        at org.elasticsearch.server@8.15.1/org.elasticsearch.plugins.PluginsService.lambda$flatMap$1(PluginsService.java:253)

See logs for more details.

ERROR: Elasticsearch did not exit normally - check the logs at /usr/bin/elasticsearch-node-frozen/elasticsearch-8.15.1/logs/elasticsearch.log

ERROR: Elasticsearch died while starting up, with exit code 1

The node has been not added. When I am trying to run it again with the same command

$ sudo -u elastic ./bin/elasticsearch --enrollment-token <enrollment_token>

I am getting an error
ERROR: Skipping security auto configuration because it appears that the node is not starting up for the first time. The node might already be part of a cluster and this auto setup utility is designed to configure Security for new clusters only., with exit code 80.

My questions are:

  1. Why does the frozen node need ca. 132 GB??? (I have only ca. 56 GB of free memory and I have no big data.)
  2. How can I resolve the problem??? (I cannot increase hard disc memory on my machine.) What can I do to re-add the frozen node with resources I have?

Thank you!

2

It pre-allocates all the space it needs for its cache, which by default is 90% of the disk. See these docs for more information, particularly about the xpack.searchable.snapshot.shared_cache.size setting.

3

I added to elasticsearch.yml of the frozen node two lines

xpack.searchable.snapshot.shared_cache.size: 8GB
xpack.searchable.snapshot.shared_cache.size.max_headroom: 4GB

I don't need here too many storage for my learning.

How can I add the node now?

The command

$ sudo -u elastic ./bin/elasticsearch --enrollment-token <enrollment_token>

gives me

ERROR: Skipping security auto configuration because it appears that the node is not starting up for the first time. The node might already be part of a cluster and this auto setup utility is designed to configure Security for new clusters only., with exit code 80

I think, I should remove this frozen node in the cluster to start it at the first time to re-add it to the cluster again... But I don't know how.

4

Yes I think it'd be simplest to start again from scratch with this node - at least, delete the contents of its data path and revert the contents of its config path back to how they were when you unpacked it.

5

When the cluster thinks that the node is added, how can I remove this node in the cluster properly?

6

I am not sure I understand the question. To remove a node from the cluster, just shut it down (i.e. kill the elasticsearch process).

7

The problem is the elasticsearch-node-frozen node is not in the list of running processes obtained with $ ps aux. For example, the other elasticsearch-node-cold node is running and I can see its ID. And I do not know, how I can re-install or re-add the the elasticsearch-node-frozen node with the changed elasticsearch.yml setting. Please help.

8

Or do you mean, I should kill the main elasticsearch process?

9

I have now these running processes related to ES.

root        6688  0.0  0.0  14980  1748 pts/0    S+   Sep27   0:00 sudo -u elastic ./bin/elasticsearch
root        6689  0.0  0.0  14980  1668 pts/1    Ss   Sep27   0:00 sudo -u elastic ./bin/elasticsearch
elastic     6690  0.0  0.4 2610228 99368 pts/1   Sl+  Sep27   2:03 /usr/bin/elasticsearch/elasticsearch-8.15.1/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=l
elastic     6751  0.4 25.3 10380980 5032696 pts/1 Sl+ Sep27  21:39 /usr/bin/elasticsearch/elasticsearch-8.15.1/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.mana
elastic     6779  0.0  0.0 108652   860 pts/1    Sl+  Sep27   0:00 /usr/bin/elasticsearch/elasticsearch-8.15.1/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root       51832  0.0  0.0  14984  2864 pts/5    S+   Sep30   0:00 sudo -u elastic ./bin/elasticsearch --enrollment-token ...
root       51833  0.0  0.0  14984  2260 pts/6    Ss   Sep30   0:00 sudo -u elastic ./bin/elasticsearch --enrollment-token ...
elastic    51834  0.0  0.7 2676804 147124 pts/6  Sl+  Sep30   0:36 /usr/bin/elasticsearch-node-cold/elasticsearch-8.15.1/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -D
elastic    51907  0.2 12.9 8080332 2579220 pts/6 Sl+  Sep30   3:06 /usr/bin/elasticsearch-node-cold/elasticsearch-8.15.1/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.sec
elastic    51937  0.0  0.0 108652  2340 pts/6    Sl+  Sep30   0:00 /usr/bin/elasticsearch-node-cold/elasticsearch-8.15.1/modules/x-pack-ml/platform/linux-x86_64/bin/controller

What should I kill? All?

10

Looks like you have two separate Elasticsearch nodes here. You can get the PIDs of the main node processes with GET _cat/nodes?h=name,ip,pid&v - that's the one to kill, it should clean up the other two.