Node not starting, even though enough disk space is now available

[hacker_21]

hi there,

I believe I recently ran out of disk space on a Node ... now that I have upgraded it's size from 1.9 to 3.9 TB, I figured it should start. But it STILL isn't starting and joining the cluster... I keep getting this error:

at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) ~[?:1.8.0_121]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_121]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_121]
	at org.apache.lucene.store.OutputStreamIndexOutput.close(OutputStreamIndexOutput.java:68) ~[lucene-core-6.4.1.jar:6.4.1 72f75b2503fa0aa4f0aff76d439874feb923bb0e - jpountz - 2017-02-01 14:43:32]
	at org.elasticsearch.gateway.MetaDataStateFormat.write(MetaDataStateFormat.java:141) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:407) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:262) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.node.Node.<init>(Node.java:238) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.3.0.jar:5.3.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.3.0.jar:5.3.0]
	Suppressed: java.io.IOException: No space left on device
		at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
		at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) ~[?:?]
		at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
		at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:?]
		at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) ~[?:?]
		at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[?:1.8.0_121]
		at java.nio.channels.Channels.writeFully(Channels.java:101) ~[?:1.8.0_121]
		at java.nio.channels.Channels.access$000(Channels.java:61) ~[?:1.8.0_121]
		at java.nio.channels.Channels$1.write(Channels.java:174) ~[?:1.8.0_121]
		at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) ~[?:1.8.0_121]
		at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_121]
		at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_121]
		at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0_121]

any ideas what i can do? Ideally I wanna restart my node or at least update it's configuration so it can scan and recognize the new 3.9 TB size of the volume!!

Any help would be greatly appreciated - this is critical data I spent weeks indexing into ES.

[warkolm (Mark Walkom)]

What OS are you on? Can you show df -h and mount if you are on linux?

[hacker_21]

This solved my problem - thanks so much Mark!!!!!!

when I ran lsblk which I was initially using to determine if the resize was successful, I would get:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 75G 0 disk /mnt
xvdc 202:32 0 75G 0 disk
xvdf 202:80 0 3.9T 0 disk

But what I didn't do was run df -h like Mark has suggested, which gave me the result:
Filesystem Size Used Avail Use% Mounted on
udev 15G 0 15G 0% /dev
tmpfs 3.0G 8.6M 3.0G 1% /run
/dev/xvda1 7.8G 1.4G 6.0G 19% /
tmpfs 15G 0 15G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 15G 0 15G 0% /sys/fs/cgroup
/dev/xvdb 74G 52M 70G 1% /mnt
/dev/xvdf 2.0T 1.9T 0 100% /home/ubuntu/mount

after looking into this difference with /dev/xvdf which is the mount point for my file system where is ES is hosted , I realized I have to resize the filesystem as well ... a simple resize of the EBS volume wasn't enough.

Eventually, I ran:
sudo resize2fs /dev/xvdf

which gave the right result for df -h:
Filesystem Size Used Avail Use% Mounted on
udev 15G 0 15G 0% /dev
tmpfs 3.0G 8.6M 3.0G 1% /run
/dev/xvda1 7.8G 1.4G 6.0G 19% /
tmpfs 15G 0 15G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 15G 0 15G 0% /sys/fs/cgroup
/dev/xvdb 74G 52M 70G 1% /mnt
tmpfs 3.0G 0 3.0G 0% /run/user/1000
/dev/xvdf 3.9T 1.9T 1.9T 50% /home/ubuntu/mount

And es booted up just fine!!! Thanks for the help again Mark