Failed to perform snapshot (index files)]; nested: FileNotFoundException

Hello
I have a 1.2.2 cluster of 6 nodes with several indexes configured with 2 to
4 replicas.
I'm trying to perform a snapshot but it completes only partially with
failures like this:

CreateSnapshotResponse[snapshotInfo=SnapshotInfo[name=2014-08-11-16-31-04,
state=PARTIAL,reason=,indices=Object[][{my_idx3,my_idx2}],
startTime=1407774870154,endTime=1407775114709,totalShards=17,
successfulShards=14,
shardFailures=Object[][{
[my_idx2][4] failed, reason [IndexShardSnapshotFailedException[[my_idx2][4]
Failed to perform snapshot (index files)]; nested: FileNotFoundException[
/home/shared_dir/indices/my_idx2/4/__0 (No such file or directory)]; ],
[my_idx2][3] failed, reason [IndexShardSnapshotFailedException[[my_idx2][3]
Failed to perform snapshot (index files)]; nested: FileNotFoundException[
/home/shared_dir/indices/my_idx2/3/__0 (No such file or directory)]; ],
[my_idx3][0] failed, reason [IndexShardSnapshotFailedException[[my_idx3][0]
Failed to perform snapshot (index files)]; nested: FileNotFoundException[
/home/shared_dir/indices/my_idx3/0/__0 (No such file or directory)]; ]}]],
headers=,remoteAddress=inet[masternode/redacted:9300]]

I used CheckIndex to verify that those shards on all nodes do not have any
problems.
What might be the reason for that?

Aleh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1c102645-3fbb-4a8d-8848-645c627a9970%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I upgraded to 1.3.1 and one of the indexes seems to backup ok now, though
the other one is still giving the error.

On Monday, August 11, 2014 8:40:44 PM UTC+3, Aleh Aleshka wrote:

Hello
I have a 1.2.2 cluster of 6 nodes with several indexes configured with 2
to 4 replicas.
I'm trying to perform a snapshot but it completes only partially with
failures like this:

CreateSnapshotResponse[snapshotInfo=SnapshotInfo[name=2014-08-11-16-31-04,
state=PARTIAL,reason=,indices=Object[{my_idx3,my_idx2}],
startTime=1407774870154,endTime=1407775114709,totalShards=17,
successfulShards=14,
shardFailures=Object[{
[my_idx2][4] failed, reason [IndexShardSnapshotFailedException[[my_idx2][4
] Failed to perform snapshot (index files)]; nested: FileNotFoundException
[/home/shared_dir/indices/my_idx2/4/__0 (No such file or directory)]; ],
[my_idx2][3] failed, reason [IndexShardSnapshotFailedException[[my_idx2][3
] Failed to perform snapshot (index files)]; nested: FileNotFoundException
[/home/shared_dir/indices/my_idx2/3/__0 (No such file or directory)]; ],
[my_idx3][0] failed, reason [IndexShardSnapshotFailedException[[my_idx3][0
] Failed to perform snapshot (index files)]; nested: FileNotFoundException
[/home/shared_dir/indices/my_idx3/0/__0 (No such file or directory)];
]}]],
headers=,remoteAddress=inet[masternode/redacted:9300]]

I used CheckIndex to verify that those shards on all nodes do not have
any problems.
What might be the reason for that?

Aleh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1385c597-94b1-4372-9c7b-a89fbd0cb4dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Well the backup is failing again on the index which was "fixed" after
upgrade.
Any idea how to debug this?

On Tuesday, August 12, 2014 7:39:42 PM UTC+3, Aleh Aleshka wrote:

I upgraded to 1.3.1 and one of the indexes seems to backup ok now, though
the other one is still giving the error.

On Monday, August 11, 2014 8:40:44 PM UTC+3, Aleh Aleshka wrote:

Hello
I have a 1.2.2 cluster of 6 nodes with several indexes configured with 2
to 4 replicas.
I'm trying to perform a snapshot but it completes only partially with
failures like this:

CreateSnapshotResponse[snapshotInfo=SnapshotInfo[name=2014-08-11-16-31-04
,state=PARTIAL,reason=,indices=Object[{my_idx3,my_idx2}],
startTime=1407774870154,endTime=1407775114709,totalShards=17,
successfulShards=14,
shardFailures=Object[{
[my_idx2][4] failed, reason [IndexShardSnapshotFailedException[[my_idx2][
4] Failed to perform snapshot (index files)]; nested:
FileNotFoundException[/home/shared_dir/indices/my_idx2/4/__0 (No such
file or directory)]; ],
[my_idx2][3] failed, reason [IndexShardSnapshotFailedException[[my_idx2][
3] Failed to perform snapshot (index files)]; nested:
FileNotFoundException[/home/shared_dir/indices/my_idx2/3/__0 (No such
file or directory)]; ],
[my_idx3][0] failed, reason [IndexShardSnapshotFailedException[[my_idx3][
0] Failed to perform snapshot (index files)]; nested:
FileNotFoundException[/home/shared_dir/indices/my_idx3/0/__0 (No such
file or directory)]; ]}]],
headers=,remoteAddress=inet[masternode/redacted:9300]]

I used CheckIndex to verify that those shards on all nodes do not have
any problems.
What might be the reason for that?

Aleh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/afbea094-d038-40ca-9f8f-dc1b1c0d9a74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.