Elasticsearch crashing with 'no space available', while disk still has 75% free storage

[Kristijan]

The elasticsearch(v5.6.12) is deployed as a single pod in a Kubernetes cluster with attached storage through a PVC.

Once every couple of months it will go into a crash loop state and complain that there is no storage left on the device, while the disk still has plenty of free storage available.

We can't quite figure out what is causing this. The only solution that works to fix this is to scale down the stateful set, remove the PVC, recreate it, and scale it back up.

We've tried increasing the storage and restarting the pod, but it makes no difference.

[2021-04-01T09:23:08,550][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.12.jar:5.6.12]
Caused by: java.lang.IllegalStateException: Failed to create node environment
	at org.elasticsearch.node.Node.<init>(Node.java:268) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.12.jar:5.6.12]
	... 6 more
Caused by: java.nio.file.FileSystemException: /usr/share/elasticsearch/data/nodes/0/_state/node-387.st.tmp: No space left on device
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
	at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) ~[?:1.8.0_181]
	at java.nio.file.Files.newOutputStream(Files.java:216) ~[?:1.8.0_181]
	at org.elasticsearch.gateway.MetaDataStateFormat.write(MetaDataStateFormat.java:124) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:419) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:263) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.12.jar:5.6.12]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.12.jar:5.6.12]
	... 6 more

[DavidTurner]

@Kristijan

No space left on device

This error message is the textual representation of the OS error code ENOSPC, so whatever is wrong is outside of Elasticsearch and you should be seeing the same error creating files by hand too. If it's not free space then it might be due to running out of inodes, or maybe some kind of quota problem.

Also 5.6 is really old, long past end of life, and you're missing out on 3½ years of new features. You should upgrade as a matter of urgency.