# Data Loss **URL:** \ **Category:** Elasticsearch\ **Created:** [February 11, 2014, 9:10pm UTC](https://discuss.elastic.co/t/data-loss/15741 "2014-02-11T21:10:28Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [February 11, 2014, 9:10pm UTC](https://discuss.elastic.co/t/data-loss/15741/1 "2014-02-11T21:10:28Z")
I've read some blogs and some email groups where users have indicated they have had data loss. In some cases user is able to recover using the source. I am wondering what are the common reasons this could happen due to ES software issue assuming there are 2+ replicas and multiple nodes available? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr%2BgDoo\_gsUbDe59-%3DpxirRpnvYgQCeD4t\_9Fqqg9tidQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr%2BgDoo_gsUbDe59-%3DpxirRpnvYgQCeD4t_9Fqqg9tidQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 12, 2014, 2:59am UTC](https://discuss.elastic.co/t/data-loss/15741/2 "2014-02-12T02:59:32Z")
Split brain would be one of the main one I can think of. Though I know some people have had issues with primary shards not initialising, though I am not sure what would cause that. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 12 February 2014 08:10, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: > I've read some blogs and some email groups where users have indicated they > have had data loss. In some cases user is able to recover using the source. > I am wondering what are the common reasons this could happen due to ES > software issue assuming there are 2+ replicas and multiple nodes available? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr%2BgDoo\_gsUbDe59-%3DpxirRpnvYgQCeD4t\_9Fqqg9tidQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr%2BgDoo_gsUbDe59-%3DpxirRpnvYgQCeD4t_9Fqqg9tidQ%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624a\_bPZ7x1YieKVsFGE2dyKn19ADFgK%2BJYbEMf9nKF7hJw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624a_bPZ7x1YieKVsFGE2dyKn19ADFgK%2BJYbEMf9nKF7hJw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Brad\_Lhotsky\_2](https://avatars.discourse-cdn.com/v4/letter/b/6de8d8/32.png) [@Brad\_Lhotsky\_2](https://discuss.elastic.co/u/Brad_Lhotsky_2)\ **Post date:** [February 12, 2014, 11:17am UTC](https://discuss.elastic.co/t/data-loss/15741/3 "2014-02-12T11:17:15Z")
There’s also an edge-case with shard allocation during cluster restarts that can result in the loss of data if a shard is being re-allocated. I saw this behaviour in 0.90.1, recently upgraded to 0.90.10 and haven’t had a failure case like this yet. My use case is logstash style daily indexes for logging data: Start: Node 1 [0] [1] Node 2 [1] [2] Node 3 [2] [0] Node 3 goes away, with allocation enabled () mean initializing: Node 1 [0] [1] (2) Node 2 [1] [2] (0) Now, Node 3 comes back, since those shards are being reallocated, it drops its copies. Node 1 [0] [1] (2) Node 2 [1] [2] (0) Node 3 — Node 2 goes away after that, allocation still enabled, Node 1 [0] [1] (2) Node 3 (0) (1) (2) There is now no full copy of shard 2, just two initialising copies which are incomplete and corrupt. When node 2 comes back into the cluster, it will discard it’s complete, uncorrupted copy of shard 2, rendering the entire index unusable, and you lose the entire index. I personally do not consider ElasticSearch a primary data store, so this isn’t an issue given the power it provides. I never expect ElasticSearch to be a reliable data store, and I think that’s OK. It has no built-in ACLs, which means even if this and the rest of the allocation related edge-cases were solved, anyone with access to port 9200 can CRUD your data. -- Brad Lhotsky On 12 Feb 2014 at 04:00:01, Mark Walkom ([markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)) wrote: Split brain would be one of the main one I can think of. Though I know some people have had issues with primary shards not initialising, though I am not sure what would cause that. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 12 February 2014 08:10, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: I've read some blogs and some email groups where users have indicated they have had data loss. In some cases user is able to recover using the source. I am wondering what are the common reasons this could happen due to ES software issue assuming there are 2+ replicas and multiple nodes available? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr%2BgDoo\_gsUbDe59-%3DpxirRpnvYgQCeD4t\_9Fqqg9tidQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr%2BgDoo_gsUbDe59-%3DpxirRpnvYgQCeD4t_9Fqqg9tidQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624a\_bPZ7x1YieKVsFGE2dyKn19ADFgK%2BJYbEMf9nKF7hJw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624a_bPZ7x1YieKVsFGE2dyKn19ADFgK%2BJYbEMf9nKF7hJw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.52fb583b.327b23c6.1375%40splitbrain.local](https://groups.google.com/d/msgid/elasticsearch/etPan.52fb583b.327b23c6.1375%40splitbrain.local). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\ **Post date:** [February 12, 2014, 3:07pm UTC](https://discuss.elastic.co/t/data-loss/15741/4 "2014-02-12T15:07:10Z")
FYI, ES has very frequent releases to fix bugs discovered by the community. If you find a data loss problem in your current install (and assuming it is indeed an ES problem), please try the latest build and see if it fixes it. Chances are it has already been discovered and fixed in the latest release. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/71fe3210-4873-417e-859a-7f83e980f2ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/71fe3210-4873-417e-859a-7f83e980f2ed%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Brad\_Lhotsky\_2](https://avatars.discourse-cdn.com/v4/letter/b/6de8d8/32.png) [@Brad\_Lhotsky\_2](https://discuss.elastic.co/u/Brad_Lhotsky_2)\ **Post date:** [February 12, 2014, 4:09pm UTC](https://discuss.elastic.co/t/data-loss/15741/5 "2014-02-12T16:09:58Z")
Appreciated, but keep in mind large installations can’t just constantly upgrade. And if ES is being used in critical infrastructure upgrading may mean many hours of recertification work with auditors and assessors. The project is still relatively young, but "just upgrade" isn’t always plausible. It takes over 2 hours for a cluster to go back to green when a single node restarts for my logging cluster. I have 15 nodes now, which means a safe upgrade path may take literally 1 working week. That assumes I can have nodes with different versions in the cluster. Or I have to lose data while I restart the _whole_ cluster, which a whole cluster restart is also ~ 4 hours. -- Brad Lhotsky On 12 Feb 2014 at 16:07:20, Binh Ly ([binh@hibalo.com](mailto:binh@hibalo.com)) wrote: ## FYI, ES has very frequent releases to fix bugs discovered by the community. If you find a data loss problem in your current install (and assuming it is indeed an ES problem), please try the latest build and see if it fixes it. Chances are it has already been discovered and fixed in the latest release. You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/71fe3210-4873-417e-859a-7f83e980f2ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/71fe3210-4873-417e-859a-7f83e980f2ed%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.52fb9cd6.2ae8944a.1375%40splitbrain.local](https://groups.google.com/d/msgid/elasticsearch/etPan.52fb9cd6.2ae8944a.1375%40splitbrain.local). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tony\_Su](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)\ **Post date:** [February 12, 2014, 7:49pm UTC](https://discuss.elastic.co/t/data-loss/15741/6 "2014-02-12T19:49:29Z")
IMO evaluating this issue starts with applying the CAP Theorem which in summary states that networked clusters with multiple nodes can offer only 2 of the following 3 desirable objectives.... Consistency Availability Partition tolerance (data distributed across nodes). ES clearly does the last two so in theory cannot guarantee the first. Of course "guarantee" is not the same as "best effort" which as expected is being done. And, this Theorem applies to multi-node cluster technologies of which ES is one. Tony On Wednesday, February 12, 2014 8:09:58 AM UTC-8, Brad Lhotsky wrote: > Appreciated, but keep in mind large installations can’t just constantly > upgrade. And if ES is being used in critical infrastructure upgrading may > mean many hours of recertification work with auditors and assessors. The > project is still relatively young, but "just upgrade" isn’t always > plausible. It takes over 2 hours for a cluster to go back to green when a > single node restarts for my logging cluster. I have 15 nodes now, which > means a safe upgrade path may take literally 1 working week. That assumes > I can have nodes with different versions in the cluster. Or I have to lose > data while I restart the _whole_ cluster, which a whole cluster restart is > also ~ 4 hours. > > -- > Brad Lhotsky > > On 12 Feb 2014 at 16:07:20, Binh Ly ([bi...@hibalo.com](mailto:bi...@hibalo.com) \) > wrote: > > ## FYI, ES has very frequent releases to fix bugs discovered by the community. If you find a data loss problem in your current install (and assuming it is indeed an ES problem), please try the latest build and see if it fixes it. Chances are it has already been discovered and fixed in the latest release. > > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/71fe3210-4873-417e-859a-7f83e980f2ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/71fe3210-4873-417e-859a-7f83e980f2ed%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f9180323-be63-454c-8ac5-2f1559f78d1b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f9180323-be63-454c-8ac5-2f1559f78d1b%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Josh\_Harrison](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@Josh\_Harrison](https://discuss.elastic.co/u/Josh_Harrison)\ **Post date:** [February 12, 2014, 8:07pm UTC](https://discuss.elastic.co/t/data-loss/15741/7 "2014-02-12T20:07:44Z")
I'm sure it isn't the case for everyone that is having data/shard problems, but I had some real trouble doing a full cluster restart on an 18 node cluster. Kinda nightmarish, actually, shards failing all over the place, lost data because of lost shards, etc. I finally realized that the gateway.recover\_after\_nodes, gateway.expected\_nodes and gateway.recover\_after\_time config properties were critical to avoiding my situation. Before the gateway configuration stuff was in there, it would take literally hours and a lot of work to get everything back to green. We dreaded a full cluster restart. After the gateway configuration stuff, a full cluster restart, from service restart on all systems to full green, takes anywhere from 2-10 minutes total. The root cause in my situation was a few nodes coming up in the cluster, and seeing a severely degraded state and trying to "fix" everything, resulting in chaos as more nodes came up. Hopefully this is helpful to someone! > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. -Josh On Wednesday, February 12, 2014 11:49:29 AM UTC-8, Tony Su wrote: > IMO evaluating this issue starts with applying the CAP Theorem which in > summary states that networked clusters with multiple nodes can offer only 2 > of the following 3 desirable objectives.... > > Consistency > Availability > Partition tolerance (data distributed across nodes). > > ES clearly does the last two so in theory cannot guarantee the first. > Of course "guarantee" is not the same as "best effort" which as expected > is being done. > And, this Theorem applies to multi-node cluster technologies of > which ES is one. > > Tony > > On Wednesday, February 12, 2014 8:09:58 AM UTC-8, Brad Lhotsky wrote: > > > Appreciated, but keep in mind large installations can’t just constantly > > upgrade. And if ES is being used in critical infrastructure upgrading may > > mean many hours of recertification work with auditors and assessors. The > > project is still relatively young, but "just upgrade" isn’t always > > plausible. It takes over 2 hours for a cluster to go back to green when a > > single node restarts for my logging cluster. I have 15 nodes now, which > > means a safe upgrade path may take literally 1 working week. That assumes > > I can have nodes with different versions in the cluster. Or I have to lose > > data while I restart the _whole_ cluster, which a whole cluster restart is > > also ~ 4 hours. > > > > -- > > Brad Lhotsky > > > > On 12 Feb 2014 at 16:07:20, Binh Ly ([bi...@hibalo.com](mailto:bi...@hibalo.com)) wrote: > > > > ## FYI, ES has very frequent releases to fix bugs discovered by the community. If you find a data loss problem in your current install (and assuming it is indeed an ES problem), please try the latest build and see if it fixes it. Chances are it has already been discovered and fixed in the latest release. > > > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/71fe3210-4873-417e-859a-7f83e980f2ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/71fe3210-4873-417e-859a-7f83e980f2ed%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c23ad042-1fdd-40da-976c-5df12a29d96b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c23ad042-1fdd-40da-976c-5df12a29d96b%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [February 12, 2014, 9:13pm UTC](https://discuss.elastic.co/t/data-loss/15741/8 "2014-02-12T21:13:04Z")
Thanks for sharing this info. It's really helpful. In any case data loss shouldn't be acceptable to anyone, especially index corruption and not able to recover at all. I think one shouldn't confuse consistency with data loss as suggested in this thread. It's also good to hear that most of the bugs around this issue is being addressed. Please let us know your findings with the new ES release. I'll look at teh gateway settings too. Do you also leave shard allocation disabled? On Wed, Feb 12, 2014 at 12:07 PM, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > I'm sure it isn't the case for everyone that is having data/shard > problems, but I had some real trouble doing a full cluster restart on an 18 > node cluster. Kinda nightmarish, actually, shards failing all over the > place, lost data because of lost shards, etc. > I finally realized that the gateway.recover\_after\_nodes, > gateway.expected\_nodes and gateway.recover\_after\_time config properties > were critical to avoiding my situation. Before the gateway configuration > stuff was in there, it would take literally hours and a lot of work to get > everything back to green. We dreaded a full cluster restart. > After the gateway configuration stuff, a full cluster restart, from > service restart on all systems to full green, takes anywhere from 2-10 > minutes total. The root cause in my situation was a few nodes coming up in > the cluster, and seeing a severely degraded state and trying to "fix" > everything, resulting in chaos as more nodes came up. > Hopefully this is helpful to someone! > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-gateway.html) > -Josh > > On Wednesday, February 12, 2014 11:49:29 AM UTC-8, Tony Su wrote: > > > IMO evaluating this issue starts with applying the CAP Theorem which in > > summary states that networked clusters with multiple nodes can offer only 2 > > of the following 3 desirable objectives.... > > > > Consistency > > Availability > > Partition tolerance (data distributed across nodes). > > > > ES clearly does the last two so in theory cannot guarantee the first. > > Of course "guarantee" is not the same as "best effort" which as expected > > is being done. > > And, this Theorem applies to multi-node cluster technologies of > > which ES is one. > > > > Tony > > > > On Wednesday, February 12, 2014 8:09:58 AM UTC-8, Brad Lhotsky wrote: > > > > > Appreciated, but keep in mind large installations can't just constantly > > > upgrade. And if ES is being used in critical infrastructure upgrading may > > > mean many hours of recertification work with auditors and assessors. The > > > project is still relatively young, but "just upgrade" isn't always > > > plausible. It takes over 2 hours for a cluster to go back to green when a > > > single node restarts for my logging cluster. I have 15 nodes now, which > > > means a safe upgrade path may take literally 1 working week. That assumes > > > I can have nodes with different versions in the cluster. Or I have to lose > > > data while I restart the _whole_ cluster, which a whole cluster restart is > > > also ~ 4 hours. > > > > > > -- > > > Brad Lhotsky > > > > > > On 12 Feb 2014 at 16:07:20, Binh Ly ([bi...@hibalo.com](mailto:bi...@hibalo.com)) wrote: > > > > > > ## FYI, ES has very frequent releases to fix bugs discovered by the community. If you find a data loss problem in your current install (and assuming it is indeed an ES problem), please try the latest build and see if it fixes it. Chances are it has already been discovered and fixed in the latest release. > > > > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/71fe3210-4873-417e-859a-7f83e980f2ed% > > > [40googlegroups.com](http://40googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/c23ad042-1fdd-40da-976c-5df12a29d96b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c23ad042-1fdd-40da-976c-5df12a29d96b%40googlegroups.com) > > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOT3TWo8mMSEc6y%3D0\_\_7JOS-kPmQoLTL0BXJ\_8Kd\_QmQT1Qg2Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOT3TWo8mMSEc6y%3D0__7JOS-kPmQoLTL0BXJ_8Kd_QmQT1Qg2Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tony\_Su](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)\ **Post date:** [February 12, 2014, 9:36pm UTC](https://discuss.elastic.co/t/data-loss/15741/9 "2014-02-12T21:36:53Z")
Josh, Your experience about recovering in only about 10 minutes is very interesting. Because my little 5-node cluster/15GB data/3500 indices is taking about an hour to recover and i know the bottleneck is the disk subsystem I'm currently on, Am curious - What is the total size data in your cluster? - how many indices? - Are the shard numbers pretty typical (5 shards per index, 1 replica for every shard) - Are you storing your data on a SAN, SCSI array or something else, and if the disks are SSD? Thx, Tony -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/26b49ca2-31a3-4800-8d76-16c6ea405c8a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/26b49ca2-31a3-4800-8d76-16c6ea405c8a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 12, 2014, 9:58pm UTC](https://discuss.elastic.co/t/data-loss/15741/10 "2014-02-12T21:58:15Z")
I use replica shard level 1, and always use latest ES version. I never had data loss, and that is also due to the fact I have access to dedicated real servers in our DC just a few meters away, and there are no servers at cloud server farms with unknown and unstable network environment. I do always update when the ES releases notes tell the update is recommended. This must be taken serious! Lucene had some nasty wipe all bugs for example, and no software is free from bugs. Updates within minor versions can take place in minutes by taking the whole cluster down and start again - but this means downtime, if that cluster is the only cluster. I tuned the nodes to recover fast. For my requirements, downtime of 15 min is acceptable. For production, you always need a staging environment. Test your updates before applying to production systems, with the same data and the same configuration. Do not let ES take care of your data alone. Make backups so you can fall back to a configuration that is known to work. Or make sure you can reindex instantly from the data source. I have also a round robin technique that works without downtime. Any operator can pull a power cable from a single server in the DC but ES will keep on running. So I can update or service the hardware (e.g. adding RAM). For updating OS and updating JVM the situation is a bit more challenging because a rolling update is not possible, but I can set up two clusters for this. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEQLWzi9yYQUhb8s2jOpswWAUDHH0DxZz9e19mfBfL9aw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEQLWzi9yYQUhb8s2jOpswWAUDHH0DxZz9e19mfBfL9aw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 12, 2014, 10:06pm UTC](https://discuss.elastic.co/t/data-loss/15741/11 "2014-02-12T22:06:45Z")
You should check index.shard.recovery.concurrent\_streams which is default to 5 per node. If you have 3500 indices, but only so few data volume, you can turn up this value very high, and your recovery will be lightning fast. But, the default value was considered carefully, so a high load of queries always has precedence over the recovery in the background, so queries do not time out that easily. Both user comfort and admin patience must be well balanced. Jörg On Wed, Feb 12, 2014 at 10:36 PM, Tony Su [tonysu999@gmail.com](mailto:tonysu999@gmail.com) wrote: > Josh, > > Your experience about recovering in only about 10 minutes is very > interesting. > Because my little 5-node cluster/15GB data/3500 indices is taking about an > hour to recover and i know the bottleneck is the disk subsystem I'm > currently on, > > Am curious > > - What is the total size data in your cluster? > - how many indices? > - Are the shard numbers pretty typical (5 shards per index, 1 replica for > every shard) > - Are you storing your data on a SAN, SCSI array or something else, and if > the disks are SSD? > > Thx, > Tony > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/26b49ca2-31a3-4800-8d76-16c6ea405c8a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/26b49ca2-31a3-4800-8d76-16c6ea405c8a%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGk0KkjBCPcuObbgn%2BvN5VoVcGg5R\_RJFyB41Std6K%3DSA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGk0KkjBCPcuObbgn%2BvN5VoVcGg5R_RJFyB41Std6K%3DSA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Josh\_Harrison](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@Josh\_Harrison](https://discuss.elastic.co/u/Josh_Harrison)\ **Post date:** [February 12, 2014, 10:23pm UTC](https://discuss.elastic.co/t/data-loss/15741/12 "2014-02-12T22:23:09Z")
This particular cluster is 16 data nodes with SSD RAIDs connected to each other and the two master nodes with infiniband. Under 100 indexes and usually 3 shards per index with 1 replica. Overall data volume is in the 1TB range. I haven't tweaked the shard allocation settings from default. -Josh On Wednesday, February 12, 2014 1:36:53 PM UTC-8, Tony Su wrote: > Josh, > > Your experience about recovering in only about 10 minutes is very > interesting. > Because my little 5-node cluster/15GB data/3500 indices is taking about an > hour to recover and i know the bottleneck is the disk subsystem I'm > currently on, > > Am curious > > - What is the total size data in your cluster? > - how many indices? > - Are the shard numbers pretty typical (5 shards per index, 1 replica for > every shard) > - Are you storing your data on a SAN, SCSI array or something else, and if > the disks are SSD? > > Thx, > Tony -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1eccb9f9-4ee7-42b3-adde-352f0c8c65d7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1eccb9f9-4ee7-42b3-adde-352f0c8c65d7%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tony\_Su](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)\ **Post date:** [February 12, 2014, 10:25pm UTC](https://discuss.elastic.co/t/data-loss/15741/13 "2014-02-12T22:25:02Z")
Jorg, Thx for the suggestion. My data is not uncommon, apache logs and according to some documentation it's advised to create a separate index for each day's logs. Tony -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c28dc0ae-2d30-418e-b44a-6cbd8917c734%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c28dc0ae-2d30-418e-b44a-6cbd8917c734%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [February 12, 2014, 10:27pm UTC](https://discuss.elastic.co/t/data-loss/15741/14 "2014-02-12T22:27:21Z")
On Wed, Feb 12, 2014 at 1:58 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > I use replica shard level 1, and always use latest ES version. I never had > data loss, and that is also due to the fact I have access to dedicated real > servers in our DC just a few meters away, and there are no servers at cloud > server farms with unknown and unstable network environment. Do you have special settings for "disable shard allocation" and "gateway" modules? Would you be able to share those settings? There are numerous number of settings and I am trying to understand which ones make most sense? How are you taking backups of data today? I believe new incremental backup feature is slated in 1.0. Are there other best practices you can share would be helpful. > I do always update when the ES releases notes tell the update is > recommended. This must be taken serious! Lucene had some nasty wipe all > bugs for example, and no software is free from bugs. > > Updates within minor versions can take place in minutes by taking the > whole cluster down and start again - but this means downtime, if that > cluster is the only cluster. I tuned the nodes to recover fast. For my > requirements, downtime of 15 min is acceptable. > > For production, you always need a staging environment. Test your updates > before applying to production systems, with the same data and the same > configuration. > > Do not let ES take care of your data alone. Make backups so you can fall > back to a configuration that is known to work. Or make sure you can reindex > instantly from the data source. > > I have also a round robin technique that works without downtime. Any > operator can pull a power cable from a single server in the DC but ES will > keep on running. So I can update or service the hardware (e.g. adding RAM). > > For updating OS and updating JVM the situation is a bit more challenging > because a rolling update is not possible, but I can set up two clusters for > this. > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEQLWzi9yYQUhb8s2jOpswWAUDHH0DxZz9e19mfBfL9aw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEQLWzi9yYQUhb8s2jOpswWAUDHH0DxZz9e19mfBfL9aw%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOT3TWoTO%3DRTuf1R-AQdiht3jTtcVdGHsJCHnb5Xx89wf\_Nqag%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOT3TWoTO%3DRTuf1R-AQdiht3jTtcVdGHsJCHnb5Xx89wf_Nqag%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [February 13, 2014, 4:56am UTC](https://discuss.elastic.co/t/data-loss/15741/15 "2014-02-13T04:56:02Z")
On Wed, Feb 12, 2014 at 1:58 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > For my requirements, downtime of 15 min is acceptable. I can only wish! I run an ecommerce site, so my requirement is no downtime. Ever. -- Ivan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDefvR3\_e4%3D\_\_5oh%3Dyn6Hcnyq9ikVMHdi0Eqyqrzh57WA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDefvR3_e4%3D__5oh%3Dyn6Hcnyq9ikVMHdi0Eqyqrzh57WA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tony\_Su](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@Tony\_Su](https://discuss.elastic.co/u/Tony_Su)\ **Post date:** [February 13, 2014, 4:54pm UTC](https://discuss.elastic.co/t/data-loss/15741/16 "2014-02-13T16:54:32Z")
Hi Mo, I've been experimenting with those settings on 1.0. The "disable shard allocation" setting doesn't seem to do anything for me. Shards still re-allocate. Which might explain why I didn't see a pre-configured setting in elasticsearch.yml so I created my own setting. But, I did find the "gateway" settings effective. After some experimentation, I decided to back off from setting "gateway.recover\_after\_nodes" from 5 to 4 on my 5 node cluster. Although requiring all nodes to be functional worked the first few times, if one node didn't join I didn't want my cluster to just sit there waiting for the node to eventually join or for me to intervene with manual action since there is no visual indication of the progress cluster nodes joining and if there is an issue it can't resolve on its own(except indirectly by using es-hq and that is only you happen to be pointing to a node that is successfully joined). Since it seems that shard re-allocation cannot be avoided entirely, I felt the time saved by allowing the cluster to start without all its nodes out-weighed trying to maximize all primary copies of the shards should be available. Tony On Wednesday, February 12, 2014 2:27:21 PM UTC-8, Mo wrote: > On Wed, Feb 12, 2014 at 1:58 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) \ \< > [joerg...@gmail.com](mailto:joerg...@gmail.com) \\> wrote: > > > I use replica shard level 1, and always use latest ES version. I never > > had data loss, and that is also due to the fact I have access to dedicated > > real servers in our DC just a few meters away, and there are no servers at > > cloud server farms with unknown and unstable network environment. > > Do you have special settings for "disable shard allocation" and "gateway" > modules? Would you be able to share those settings? There are numerous > number of settings and I am trying to understand which ones make most sense? > > How are you taking backups of data today? I believe new incremental backup > feature is slated in 1.0. > > Are there other best practices you can share would be helpful. > > > I do always update when the ES releases notes tell the update is > > recommended. This must be taken serious! Lucene had some nasty wipe all > > bugs for example, and no software is free from bugs. > > > > Updates within minor versions can take place in minutes by taking the > > whole cluster down and start again - but this means downtime, if that > > cluster is the only cluster. I tuned the nodes to recover fast. For my > > requirements, downtime of 15 min is acceptable. > > > > For production, you always need a staging environment. Test your updates > > before applying to production systems, with the same data and the same > > configuration. > > > > Do not let ES take care of your data alone. Make backups so you can fall > > back to a configuration that is known to work. Or make sure you can reindex > > instantly from the data source. > > > > I have also a round robin technique that works without downtime. Any > > operator can pull a power cable from a single server in the DC but ES will > > keep on running. So I can update or service the hardware (e.g. adding RAM). > > > > For updating OS and updating JVM the situation is a bit more challenging > > because a rolling update is not possible, but I can set up two clusters for > > this. > > > > Jörg > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEQLWzi9yYQUhb8s2jOpswWAUDHH0DxZz9e19mfBfL9aw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEQLWzi9yYQUhb8s2jOpswWAUDHH0DxZz9e19mfBfL9aw%40mail.gmail.com) > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e5a631c1-f776-48e7-b7f3-631ab02b7cf4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e5a631c1-f776-48e7-b7f3-631ab02b7cf4%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 13, 2014, 10:09pm UTC](https://discuss.elastic.co/t/data-loss/15741/17 "2014-02-13T22:09:17Z")
There are changes with 1.0.0 - > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. cluster.routing.allocation.disable\_allocation has been deprecated for cluster.routing.allocation.enable Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 14 February 2014 03:54, Tony Su [tonysu999@gmail.com](mailto:tonysu999@gmail.com) wrote: > Hi Mo, > I've been experimenting with those settings on 1.0. > The "disable shard allocation" setting doesn't seem to do anything for me. > Shards still re-allocate. Which might explain why I didn't see a > pre-configured setting in elasticsearch.yml so I created my own setting. > > But, I did find the "gateway" settings effective. > After some experimentation, I decided to back off from setting > "gateway.recover\_after\_nodes" from 5 to 4 on my 5 node cluster. Although > requiring all nodes to be functional worked the first few times, if one > node didn't join I didn't want my cluster to just sit there waiting for the > node to eventually join or for me to intervene with manual action since > there is no visual indication of the progress cluster nodes joining and if > there is an issue it can't resolve on its own(except indirectly by using > es-hq and that is only you happen to be pointing to a node that is > successfully joined). Since it seems that shard re-allocation cannot be > avoided entirely, I felt the time saved by allowing the cluster to start > without all its nodes out-weighed trying to maximize all primary copies of > the shards should be available. > > Tony > > On Wednesday, February 12, 2014 2:27:21 PM UTC-8, Mo wrote: > > > On Wed, Feb 12, 2014 at 1:58 PM, [joerg...@gmail.com](mailto:joerg...@gmail.com) [joerg...@gmail.com](mailto:joerg...@gmail.com)wrote: > > > > > I use replica shard level 1, and always use latest ES version. I never > > > had data loss, and that is also due to the fact I have access to dedicated > > > real servers in our DC just a few meters away, and there are no servers at > > > cloud server farms with unknown and unstable network environment. > > > > Do you have special settings for "disable shard allocation" and "gateway" > > modules? Would you be able to share those settings? There are numerous > > number of settings and I am trying to understand which ones make most sense? > > > > How are you taking backups of data today? I believe new incremental > > backup feature is slated in 1.0. > > > > Are there other best practices you can share would be helpful. > > > > > I do always update when the ES releases notes tell the update is > > > recommended. This must be taken serious! Lucene had some nasty wipe all > > > bugs for example, and no software is free from bugs. > > > > > > Updates within minor versions can take place in minutes by taking the > > > whole cluster down and start again - but this means downtime, if that > > > cluster is the only cluster. I tuned the nodes to recover fast. For my > > > requirements, downtime of 15 min is acceptable. > > > > > > For production, you always need a staging environment. Test your updates > > > before applying to production systems, with the same data and the same > > > configuration. > > > > > > Do not let ES take care of your data alone. Make backups so you can fall > > > back to a configuration that is known to work. Or make sure you can reindex > > > instantly from the data source. > > > > > > I have also a round robin technique that works without downtime. Any > > > operator can pull a power cable from a single server in the DC but ES will > > > keep on running. So I can update or service the hardware (e.g. adding RAM). > > > > > > For updating OS and updating JVM the situation is a bit more challenging > > > because a rolling update is not possible, but I can set up two clusters for > > > this. > > > > > > Jörg > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/CAKdsXoEQLWzi9yYQUhb8s2jOpswWA > > > UDHH0DxZz9e19mfBfL9aw%[40mail.gmail.com](http://40mail.gmail.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/e5a631c1-f776-48e7-b7f3-631ab02b7cf4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e5a631c1-f776-48e7-b7f3-631ab02b7cf4%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bg30MvKHUuyY%3DR4vpiwyDgED%3DDuhjdmj16\_fCc5FkWtA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bg30MvKHUuyY%3DR4vpiwyDgED%3DDuhjdmj16_fCc5FkWtA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 13, 2014, 10:20pm UTC](https://discuss.elastic.co/t/data-loss/15741/18 "2014-02-13T22:20:49Z")
I'm keeping my shards small (1-5 GB), I do not disable shard allocation, the recovery is fast enough (a few minutes) Gateway is local, the default. I keep backups on the file system, with plain cp -a or rsync, but also on the data source side - I can reindex everything in a few hours, I have only ~100 mio. docs to index. Because I don't operate a shared file system, I can not use ES 1.0 snapshot/restore yet. Jörg On Wed, Feb 12, 2014 at 11:27 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote: > Do you have special settings for "disable shard allocation" and "gateway" > modules? Would you be able to share those settings? There are numerous > number of settings and I am trying to understand which ones make most sense? > > How are you taking backups of data today? I believe new incremental backup > feature is slated in 1.0. > > Are there other best practices you can share would be helpful. > > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEE-XtuDbDnv8CLrJ\_tDxDLB7spZ-\_cfcpjaAa4\_qdB4Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEE-XtuDbDnv8CLrJ_tDxDLB7spZ-_cfcpjaAa4_qdB4Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 13, 2014, 10:26pm UTC](https://discuss.elastic.co/t/data-loss/15741/19 "2014-02-13T22:26:59Z")
Personally this is what I am after because I know how frustrating it is for the user. For operating library catalog search, we do not lose (too much) money if one of our systems go down for some minutes, but users lose time and get annoyed. Hard to convince the supervisors to set up 24/7 operation mode and SLAs - this is more expensive than random short system outages. And, even with 24/7, the search engine is only one part of a whole complex network. Jörg On Thu, Feb 13, 2014 at 5:56 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > On Wed, Feb 12, 2014 at 1:58 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > For my requirements, downtime of 15 min is acceptable. > > I can only wish! I run an ecommerce site, so my requirement is no > downtime. Ever. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF-hF%3D6q%3DcjihZ6qksCV6sEmQ8nTONkQhLjW1AAR7YkHw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF-hF%3D6q%3DcjihZ6qksCV6sEmQ8nTONkQhLjW1AAR7YkHw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [February 13, 2014, 10:33pm UTC](https://discuss.elastic.co/t/data-loss/15741/20 "2014-02-13T22:33:41Z")
This is interesting, I was thinking by having more replicas we can work around issues when few nodes go down. Is this more challenging than that? I am trying to understand where should I put more focus on to get HA search solution. We have large data sets so I also worry about too much index movement causing performance issues. Is there a downside of completely disabling shard allocation. On Thu, Feb 13, 2014 at 2:26 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > Personally this is what I am after because I know how frustrating it is > for the user. For operating library catalog search, we do not lose (too > much) money if one of our systems go down for some minutes, but users lose > time and get annoyed. Hard to convince the supervisors to set up 24/7 > operation mode and SLAs - this is more expensive than random short system > outages. And, even with 24/7, the search engine is only one part of a whole > complex network. > > Jörg > > On Thu, Feb 13, 2014 at 5:56 AM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > On Wed, Feb 12, 2014 at 1:58 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > For my requirements, downtime of 15 min is acceptable. > > > > I can only wish! I run an ecommerce site, so my requirement is no > > downtime. Ever. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF-hF%3D6q%3DcjihZ6qksCV6sEmQ8nTONkQhLjW1AAR7YkHw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoF-hF%3D6q%3DcjihZ6qksCV6sEmQ8nTONkQhLjW1AAR7YkHw%40mail.gmail.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOT3TWqUcHo3sqb-aUXwp6zJ5-fy-PUE1hy\_r6pgoO2Rh1w51w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOT3TWqUcHo3sqb-aUXwp6zJ5-fy-PUE1hy_r6pgoO2Rh1w51w%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). [Next page](https://discuss.elastic.co/t/data-loss/15741.md?page=2)