# Lost the data **URL:** \ **Category:** Elasticsearch\ **Created:** [April 19, 2012, 3:15pm UTC](https://discuss.elastic.co/t/lost-the-data/7400 "2012-04-19T15:15:12Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Amit\_Mohan](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@Amit\_Mohan](https://discuss.elastic.co/u/Amit_Mohan)\ **Post date:** [April 19, 2012, 3:15pm UTC](https://discuss.elastic.co/t/lost-the-data/7400/1 "2012-04-19T15:15:12Z")
I am running 34 shards on 17 heavy duty machines. Due to a network maint. we had to restart the cluster ( 0.19.1 ). After the restart 1 of the machine lost all it's shard data for some reason. There is nothing in the logs. No disk/memory/hardware issue detected. Just the data is gone. Now I can't bring the cluster to Green. Is there a way to tell the cluster that it should ignore those 2 shards' data and come up happily and start indexing again ? I am ready to loose those 2 shards' data since it's the logs I am indexing there. Thanks for any help! -Amit Mohan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 19, 2012, 3:19pm UTC](https://discuss.elastic.co/t/lost-the-data/7400/2 "2012-04-19T15:19:05Z")
Hi, elasticsearch won't delete the data explicitly, I wonder though, do you have replicas for the shards setup? How many indices do you have where those 2 shards are missing now? On Thu, Apr 19, 2012 at 6:15 PM, Amit Mohan [amisakrenvan@gmail.com](mailto:amisakrenvan@gmail.com) wrote: > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > machine lost all it's shard data for some reason. There is nothing in the > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > can't bring the cluster to Green. > > Is there a way to tell the cluster that it should ignore those 2 shards' > data and come up happily and start indexing again ? I am ready to loose > those 2 shards' data since it's the logs I am indexing there. > > Thanks for any help! > -Amit Mohan ---
**Author:** ![Amit\_Mohan](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@Amit\_Mohan](https://discuss.elastic.co/u/Amit_Mohan)\ **Post date:** [April 19, 2012, 3:25pm UTC](https://discuss.elastic.co/t/lost-the-data/7400/3 "2012-04-19T15:25:16Z")
I had no replica in order to save some space. I have 17 machines and 34 shards, so that machine had 2 shards on it, ( 0 and 24 ). All other 32 shards are good though. I had no gateway setup either which I intend to do now but not sure in this case it would have helped. On Thursday, April 19, 2012 11:19:05 AM UTC-4, kimchy wrote: > Hi, > > elasticsearch won't delete the data explicitly, I wonder though, do you > have replicas for the shards setup? How many indices do you have where > those 2 shards are missing now? > > On Thu, Apr 19, 2012 at 6:15 PM, Amit Mohan [amisakrenvan@gmail.com](mailto:amisakrenvan@gmail.com)wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan On Thursday, April 19, 2012 11:19:05 AM UTC-4, kimchy wrote: > Hi, > > elasticsearch won't delete the data explicitly, I wonder though, do you > have replicas for the shards setup? How many indices do you have where > those 2 shards are missing now? > > On Thu, Apr 19, 2012 at 6:15 PM, Amit Mohan [amisakrenvan@gmail.com](mailto:amisakrenvan@gmail.com)wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan On Thursday, April 19, 2012 11:19:05 AM UTC-4, kimchy wrote: > Hi, > > elasticsearch won't delete the data explicitly, I wonder though, do you > have replicas for the shards setup? How many indices do you have where > those 2 shards are missing now? > > On Thu, Apr 19, 2012 at 6:15 PM, Amit Mohan [amisakrenvan@gmail.com](mailto:amisakrenvan@gmail.com)wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan On Thursday, April 19, 2012 11:19:05 AM UTC-4, kimchy wrote: > Hi, > > elasticsearch won't delete the data explicitly, I wonder though, do you > have replicas for the shards setup? How many indices do you have where > those 2 shards are missing now? > > On Thu, Apr 19, 2012 at 6:15 PM, Amit Mohan [amisakrenvan@gmail.com](mailto:amisakrenvan@gmail.com)wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan ---
**Author:** ![Amit\_Mohan](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@Amit\_Mohan](https://discuss.elastic.co/u/Amit_Mohan)\ **Post date:** [April 19, 2012, 3:28pm UTC](https://discuss.elastic.co/t/lost-the-data/7400/4 "2012-04-19T15:28:22Z")
Sorry, did not read your question properly. I have only 1 index. On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > machine lost all it's shard data for some reason. There is nothing in the > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > can't bring the cluster to Green. > > Is there a way to tell the cluster that it should ignore those 2 shards' > data and come up happily and start indexing again ? I am ready to loose > those 2 shards' data since it's the logs I am indexing there. > > Thanks for any help! > -Amit Mohan On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > machine lost all it's shard data for some reason. There is nothing in the > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > can't bring the cluster to Green. > > Is there a way to tell the cluster that it should ignore those 2 shards' > data and come up happily and start indexing again ? I am ready to loose > those 2 shards' data since it's the logs I am indexing there. > > Thanks for any help! > -Amit Mohan On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > machine lost all it's shard data for some reason. There is nothing in the > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > can't bring the cluster to Green. > > Is there a way to tell the cluster that it should ignore those 2 shards' > data and come up happily and start indexing again ? I am ready to loose > those 2 shards' data since it's the logs I am indexing there. > > Thanks for any help! > -Amit Mohan On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > machine lost all it's shard data for some reason. There is nothing in the > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > can't bring the cluster to Green. > > Is there a way to tell the cluster that it should ignore those 2 shards' > data and come up happily and start indexing again ? I am ready to loose > those 2 shards' data since it's the logs I am indexing there. > > Thanks for any help! > -Amit Mohan On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > machine lost all it's shard data for some reason. There is nothing in the > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > can't bring the cluster to Green. > > Is there a way to tell the cluster that it should ignore those 2 shards' > data and come up happily and start indexing again ? I am ready to loose > those 2 shards' data since it's the logs I am indexing there. > > Thanks for any help! > -Amit Mohan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 19, 2012, 3:31pm UTC](https://discuss.elastic.co/t/lost-the-data/7400/5 "2012-04-19T15:31:33Z")
No need to setup a gateway, the local gateway is good and the way you get high availability is by using replicas. You can't bring back those 2 shards, or force them to be created easily. What you can do is create another index and index new data to it, and use the old index just to search (you can use aliases to search over both indexes). Search on the index that does not have all shards will still work, indexing operations that will end up being routed to the 2 missing shards will fail. On Thu, Apr 19, 2012 at 6:28 PM, Amit Mohan [amisakrenvan@gmail.com](mailto:amisakrenvan@gmail.com) wrote: > Sorry, did not read your question properly. I have only 1 index. > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan ---
**Author:** ![Amit\_Mohan](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@Amit\_Mohan](https://discuss.elastic.co/u/Amit_Mohan)\ **Post date:** [April 19, 2012, 4:37pm UTC](https://discuss.elastic.co/t/lost-the-data/7400/6 "2012-04-19T16:37:11Z")
Thanks for your. That is what I am going to do then until the old data becomes irrelevant. Thanks again! On Thursday, April 19, 2012 11:31:33 AM UTC-4, kimchy wrote: > No need to setup a gateway, the local gateway is good and the way you get > high availability is by using replicas. You can't bring back those 2 > shards, or force them to be created easily. What you can do is create > another index and index new data to it, and use the old index just to > search (you can use aliases to search over both indexes). Search on the > index that does not have all shards will still work, indexing operations > that will end up being routed to the 2 missing shards will fail. > > On Thu, Apr 19, 2012 at 6:28 PM, Amit Mohan [amisakrenvan@gmail.com](mailto:amisakrenvan@gmail.com)wrote: > > > Sorry, did not read your question properly. I have only 1 index. > > > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > > > I am running 34 shards on 17 heavy duty machines. Due to a network > > > maint. we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > > machine lost all it's shard data for some reason. There is nothing in the > > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > > can't bring the cluster to Green. > > > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > > data and come up happily and start indexing again ? I am ready to loose > > > those 2 shards' data since it's the logs I am indexing there. > > > > > > Thanks for any help! > > > -Amit Mohan > > > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > > > I am running 34 shards on 17 heavy duty machines. Due to a network > > > maint. we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > > machine lost all it's shard data for some reason. There is nothing in the > > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > > can't bring the cluster to Green. > > > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > > data and come up happily and start indexing again ? I am ready to loose > > > those 2 shards' data since it's the logs I am indexing there. > > > > > > Thanks for any help! > > > -Amit Mohan > > > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > > > I am running 34 shards on 17 heavy duty machines. Due to a network > > > maint. we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > > machine lost all it's shard data for some reason. There is nothing in the > > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > > can't bring the cluster to Green. > > > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > > data and come up happily and start indexing again ? I am ready to loose > > > those 2 shards' data since it's the logs I am indexing there. > > > > > > Thanks for any help! > > > -Amit Mohan > > > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > > > I am running 34 shards on 17 heavy duty machines. Due to a network > > > maint. we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > > machine lost all it's shard data for some reason. There is nothing in the > > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > > can't bring the cluster to Green. > > > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > > data and come up happily and start indexing again ? I am ready to loose > > > those 2 shards' data since it's the logs I am indexing there. > > > > > > Thanks for any help! > > > -Amit Mohan > > > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > > > I am running 34 shards on 17 heavy duty machines. Due to a network > > > maint. we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > > machine lost all it's shard data for some reason. There is nothing in the > > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > > can't bring the cluster to Green. > > > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > > data and come up happily and start indexing again ? I am ready to loose > > > those 2 shards' data since it's the logs I am indexing there. > > > > > > Thanks for any help! > > > -Amit Mohan ---
**Author:** ![james0tempe](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@james0tempe](https://discuss.elastic.co/u/james0tempe)\ **Post date:** [July 22, 2012, 9:45am UTC](https://discuss.elastic.co/t/lost-the-data/7400/7 "2012-07-22T09:45:21Z")
If there's a "hard way" to replace a failed shard, it would be very useful to us. Scenario: A five node cluster with 5 shards, 5 replicas, loses shard 2 and 3 and cannot recover. We would like to create two new empty shards to replace the lost ones, and deal with the data loss for the sake of cluster integrity. Would the following idea work? Create another 5 shard index with the same name, somewhere else, and then copy the directory and metadata of the replacement shards in place of the broken shards. Is there something we can do to the metadata that would make that a workable plan? - James No need to setup a gateway, the local gateway is good and the way you get high availability is by using replicas. You can't bring back those 2 shards, or **force them to be created easily**. What you can do is create another index and index new data to it, and use the old index just to search (you can use aliases to search over both indexes). Search on the index that does not have all shards will still work, indexing operations that will end up being routed to the 2 missing shards will fail. On Thu, Apr 19, 2012 at 6:28 PM, Amit Mohan \ wrote: > Sorry, did not read your question properly. I have only 1 index. > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan > > On Thursday, April 19, 2012 11:15:12 AM UTC-4, Amit Mohan wrote: > > > I am running 34 shards on 17 heavy duty machines. Due to a network maint. > > we had to restart the cluster ( 0.19.1 ). After the restart 1 of the > > machine lost all it's shard data for some reason. There is nothing in the > > logs. No disk/memory/hardware issue detected. Just the data is gone. Now I > > can't bring the cluster to Green. > > > > Is there a way to tell the cluster that it should ignore those 2 shards' > > data and come up happily and start indexing again ? I am ready to loose > > those 2 shards' data since it's the logs I am indexing there. > > > > Thanks for any help! > > -Amit Mohan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:19am UTC](https://discuss.elastic.co/t/lost-the-data/7400/8 "2017-07-06T03:19:31Z")