# Recovery using different storage location than expected for? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 7, 2011, 9:15pm UTC](https://discuss.elastic.co/t/recovery-using-different-storage-location-than-expected-for/5801 "2011-11-07T21:15:00Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![jminard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jminard/32/1848_2.png) [@jminard](https://discuss.elastic.co/u/jminard)\ **Post date:** [November 7, 2011, 9:15pm UTC](https://discuss.elastic.co/t/recovery-using-different-storage-location-than-expected-for/5801/1 "2011-11-07T21:15:00Z")
I have a 5 node EC2 setup, index+logs+work is on the ephemeral storage at /data, which eleasticsearch binaries are started from a smaller EBS boot volume. During recovery I get an error on recovery that the file is too big to transfer due to running out of disk space, but where is it writing this chunk since the /data directory has plenty of space. Config: index: store: type: niofs number\_of\_shards: 15 number\_of\_replicas: 2 path: data: /data/elasticsearch/data work: /data/elasticsearch/work logs: /data/elasticsearch/logs gateway: type: s3 s3: bucket: nameremoved-elasticsearch recovery\_after\_nodes: 3 recovery\_after\_time: 5m expected\_nodes: 5 plugins: [mobz/elasticsearch-head, lukas-vlcek/bigdesk] discovery: type: ec2 cloud: aws: access\_key: secret\_key: cluster: name: elasticsearch Log output: [2011-11-07 21:08:34,274][WARN][cluster.action.shard] [Banshee] received shard failed for [media][8], node[\_Swg\_4NsRru28Ppp3\_1mFg], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [media][8]: Recovery failed from [Banshee][7U-Kn7aASQy1TGPEo0aWew][inet[/10.214.137.156:9300]] into [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]]]; nested: RemoteTransportException[[Banshee][inet[/10.214.137.156:9300]] [index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media][8] Phase[1] Execution failed]; nested: RecoverFilesRecoveryException[[media][8] Failed to transfer [200] files with total size of [532.5mb]]; nested: RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/ shard/recovery/fileChunk]]; nested: IOException[No space left on device]; ]] [2011-11-07 21:08:34,804][WARN][cluster.action.shard] [Banshee] received shard failed for [media][11], node[\_Swg\_4NsRru28Ppp3\_1mFg], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [media][11]: Recovery failed from [Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]] into [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]]]; nested: RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]] [index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media][11] Phase[1] Execution failed]; nested: RecoverFilesRecoveryException[[media][11] Failed to transfer [190] files with total size of [537mb]]; nested: RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/ shard/recovery/fileChunk]]; nested: IOException[No space left on device]; ]] ^C jminard@domU-12-31-39-0B-86-52:/data/elasticsearch/logs$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.9G 7.4G 172M 98% / none 3.7G 112K 3.7G 1% /dev none 3.8G 0 3.8G 0% /dev/shm none 3.8G 52K 3.8G 1% /var/run none 3.8G 0 3.8G 0% /var/lock none 3.8G 0 3.8G 0% /lib/init/rw /dev/sdb 414G 199M 393G 1% /mnt ---
**Author:** ![Tomasz\_Kloc](https://avatars.discourse-cdn.com/v4/letter/t/958977/32.png) [@Tomasz\_Kloc](https://discuss.elastic.co/u/Tomasz_Kloc)\ **Post date:** [November 7, 2011, 9:44pm UTC](https://discuss.elastic.co/t/recovery-using-different-storage-location-than-expected-for/5801/2 "2011-11-07T21:44:48Z")
Hi On 07.11.2011 22:15, jminard wrote: > jminard@domU-12-31-39-0B-86-52:/data/elasticsearch/logs$ df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda1 7.9G 7.4G 172M 98% / > none 3.7G 112K 3.7G 1% /dev > none 3.8G 0 3.8G 0% /dev/shm > none 3.8G 52K 3.8G 1% /var/run > none 3.8G 0 3.8G 0% /var/lock > none 3.8G 0 3.8G 0% /lib/init/rw > /dev/sdb 414G 199M 393G 1% /mnt it seems that /data is mounted on /dev/sda1 (98% disk usage) to be sure check df -h /data ---
**Author:** ![jminard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jminard/32/1848_2.png) [@jminard](https://discuss.elastic.co/u/jminard)\ **Post date:** [November 7, 2011, 10:05pm UTC](https://discuss.elastic.co/t/recovery-using-different-storage-location-than-expected-for/5801/3 "2011-11-07T22:05:05Z")
Arg, /data was to be created as a symlink to the /mnt directory, but the script ran the command with the wrong permissions. Thanks for pointing out the obvious. On Nov 7, 1:44 pm, Tomasz Kloc [tomek.kloc....@gmail.com](mailto:tomek.kloc....@gmail.com) wrote: > Hi > > On 07.11.2011 22:15, jminard wrote:\> jminard@domU-12-31-39-0B-86-52:/data/elasticsearch/logs$ df -h > > > Filesystem Size Used Avail Use% Mounted on > > /dev/sda1 7.9G 7.4G 172M 98% / > > none 3.7G 112K 3.7G 1% /dev > > none 3.8G 0 3.8G 0% /dev/shm > > none 3.8G 52K 3.8G 1% /var/run > > none 3.8G 0 3.8G 0% /var/lock > > none 3.8G 0 3.8G 0% /lib/init/rw > > /dev/sdb 414G 199M 393G 1% /mnt > > it seems that /data is mounted on /dev/sda1 (98% disk usage) > > to be sure check df -h /data ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:49am UTC](https://discuss.elastic.co/t/recovery-using-different-storage-location-than-expected-for/5801/4 "2017-07-06T03:49:43Z")