# Shard error: listed commit\_point \[commit-bx\]/\[429\], but not all files exists, ignoring **URL:** \ **Category:** Elasticsearch\ **Created:** [August 13, 2013, 6:19pm UTC](https://discuss.elastic.co/t/shard-error-listed-commit-point-commit-bx-429-but-not-all-files-exists-ignoring/13203 "2013-08-13T18:19:22Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Oli\_McCormack](https://avatars.discourse-cdn.com/v4/letter/o/fbc32d/32.png) [@Oli\_McCormack](https://discuss.elastic.co/u/Oli_McCormack)\ **Post date:** [August 13, 2013, 6:19pm UTC](https://discuss.elastic.co/t/shard-error-listed-commit-point-commit-bx-429-but-not-all-files-exists-ignoring/13203/1 "2013-08-13T18:19:22Z")
Hi all, Yesterday afternoon I started to see a cluster red state and the following exception: [2013-08-12 23:59:59,587][WARN][cluster.action.shard] [hostname] sending failed shard for [einstein][6], node[w3Ux3oepSrWr5tP\_cGWbBQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[einstein][6] No commit point data is available in gateway]]] [2013-08-12 23:59:59,904][WARN][index.gateway.s3] [hostname] [einstein][6] _listed commit\_point [commit-bx]/[429], but not all files exists, ignoring_ [2013-08-12 23:59:59,904][WARN][indices.cluster] [hostname] [einstein][6] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [einstein][6] No commit point data is available in gateway at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:427) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) This is the first time it has ever happened. If I actually look at the stored information for that shard, I see content like this: s3.gateway.bucket/elasticsearch/indices/einstein/6/... (many files) s3.gateway.bucket/elasticsearch/indices/einstein/6/\_\_2d6 s3.gateway.bucket/elasticsearch/indices/einstein/6/\_\_2dj s3.gateway.bucket/elasticsearch/indices/einstein/6/commit-bx What I also notice is that the commit-bx (a) doesn't have an entry for \_\_2dj and (b) does have an entry for \_\_2di, but \_\_2di does not exist in the directory listing and (c) all other files seem to be accounted for. Is there any way for me to recover from this case? Or even to get the documents that failed to be indexed, so that I could clear the index, but still have the source docs and re-ingest them? I've seen other cases[http://elasticsearch-users.115913.n3.nabble.com/No-commit-point-data-is-available-in-gateway-td3727782.html](http://elasticsearch-users.115913.n3.nabble.com/No-commit-point-data-is-available-in-gateway-td3727782.html)on the list that suggest that the index now needs to be completely deleted/replaced, but at least being able to get at the documents would be useful. Thank you for any pointers, I'll report back if I make any progress, oli -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Oli\_McCormack](https://avatars.discourse-cdn.com/v4/letter/o/fbc32d/32.png) [@Oli\_McCormack](https://discuss.elastic.co/u/Oli_McCormack)\ **Post date:** [August 13, 2013, 6:42pm UTC](https://discuss.elastic.co/t/shard-error-listed-commit-point-commit-bx-429-but-not-all-files-exists-ignoring/13203/2 "2013-08-13T18:42:57Z")
I actually just managed to get my cluster back to a green state. The resolution was very odd (I'm still not sure if \*data \*is resolved, but the cluster seems happy). Here's what I observed: The content length of \_\_2dj was the same as what commit-bx had for the \_\_2di entry: _\_\_2dj:_ _1422_ s3.gateway.bucket/elasticsearch/indices/einstein/6/\_\_2dj _commit-bx_ "\_\_2di" : { "physical\_name" : "segments\_3h", "length" : _1422_ }, I guessed that these may be referring to the same thing, just that how names got assigned was weird, so I decided to try and rename the \_\_2dj file to \_\_2di, to see if commit-bx would be happier. I did this, which prompted _something_ to then rename the file back to \_\_2dj on my behalf, but which also seemed to replace the \_\_2di entry in commit-bx with an equivalent \_\_2dj entry. It also bumped the version number in commit-bx from 429 -\> 430. As mentioned, the cluster now seems happy, but I need to verify if there are any problems or loss. Anyone with any insight into what may have happened, or if the current state is ok, I'd be very interested to hear what you have to say! Thanks, oli On Tuesday, August 13, 2013 11:19:22 AM UTC-7, Oli wrote: > Hi all, > > Yesterday afternoon I started to see a cluster red state and the following > exception: > > [2013-08-12 23:59:59,587][WARN][cluster.action.shard] [hostname] > sending failed shard for [einstein][6], node[w3Ux3oepSrWr5tP\_cGWbBQ], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[einstein][6] No commit point data is > available in gateway]]] > [2013-08-12 23:59:59,904][WARN][index.gateway.s3] [hostname] > [einstein][6] _listed commit\_point [commit-bx]/[429], but not all files > exists, ignoring_ > [2013-08-12 23:59:59,904][WARN][indices.cluster] [hostname] > [einstein][6] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [einstein][6] No commit point data is available in gateway > at > org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:427) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > This is the first time it has ever happened. If I actually look at the > stored information for that shard, I see content like this: > s3.gateway.bucket/elasticsearch/indices/einstein/6/... (many files) > s3.gateway.bucket/elasticsearch/indices/einstein/6/\_\_2d6 > s3.gateway.bucket/elasticsearch/indices/einstein/6/\_\_2dj > s3.gateway.bucket/elasticsearch/indices/einstein/6/commit-bx > > What I also notice is that the commit-bx (a) doesn't have an entry for > \_\_2dj and (b) does have an entry for \_\_2di, but \_\_2di does not exist in the > directory listing and (c) all other files seem to be accounted for. > > Is there any way for me to recover from this case? Or even to get the > documents that failed to be indexed, so that I could clear the index, but > still have the source docs and re-ingest them? > > I've seen other cases[http://elasticsearch-users.115913.n3.nabble.com/No-commit-point-data-is-available-in-gateway-td3727782.html](http://elasticsearch-users.115913.n3.nabble.com/No-commit-point-data-is-available-in-gateway-td3727782.html)on the list that suggest that the index now needs to be completely > deleted/replaced, but at least being able to get at the documents would be > useful. > > Thank you for any pointers, I'll report back if I make any progress, > oli -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 13, 2013, 8:44pm UTC](https://discuss.elastic.co/t/shard-error-listed-commit-point-commit-bx-429-but-not-all-files-exists-ignoring/13203/3 "2013-08-13T20:44:10Z")
Not for the faint hearted. Congrats! Can't help, but this smells like the JVM was unable to use file descriptors to write stuff out and failed silently. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Oli\_McCormack](https://avatars.discourse-cdn.com/v4/letter/o/fbc32d/32.png) [@Oli\_McCormack](https://discuss.elastic.co/u/Oli_McCormack)\ **Post date:** [August 14, 2013, 1:12am UTC](https://discuss.elastic.co/t/shard-error-listed-commit-point-commit-bx-429-but-not-all-files-exists-ignoring/13203/4 "2013-08-14T01:12:25Z")
Thanks Jörg, To follow up, I confirmed that the index appears untainted. I probably got lucky in that the index was extremely low traffic, so even just investigating the files maintained by the index was reasonably parseable. It's still unclear to me if the raw docs themselves are written somewhere that I could recover them (even if I had to rebuild the index, knowing I hadn't lost docs would be excellent!). If I find out, I'll post back. - oli On Tuesday, August 13, 2013 1:44:10 PM UTC-7, Jörg Prante wrote: > Not for the faint hearted. Congrats! > > Can't help, but this smells like the JVM was unable to use file > descriptors to write stuff out and failed silently. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:21am UTC](https://discuss.elastic.co/t/shard-error-listed-commit-point-commit-bx-429-but-not-all-files-exists-ignoring/13203/5 "2017-07-06T02:21:14Z")