# Reindexing and error recovery **URL:** \ **Category:** Elasticsearch\ **Created:** [July 26, 2011, 12:34pm UTC](https://discuss.elastic.co/t/reindexing-and-error-recovery/4965 "2011-07-26T12:34:50Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [July 26, 2011, 12:34pm UTC](https://discuss.elastic.co/t/reindexing-and-error-recovery/4965/1 "2011-07-26T12:34:50Z")
Excuse me in advance for the long post. Up until now, I have avoided rebuilding from source but now is the time to finally dive in. Trying to determine reindexing best practices. Using aliases appears to be the suggested solution to creating updated indices in parallel to searching an existing index. Searchers always use the same index, while the indexers create new indexes whenever a reindex is needed, which is then aliased to. While the new index is being created, do you stop indexing on the existing index? Scan acts like a cursor, so new inserts after the scan was created do not get picked up. Does anybody use a river and reindex data? A river is associated with an index at creation. If the above recipe holds true, I need to delete the river and then create a new one with the updated index name. Not the best scenario since I will lose updates, but the data is not mission critical. On the topic of reindexing, I also need to recover data from an incorrect node. I switched a single-node elasticsearch server from using monit to using the service wrapper. On the same day that I made the migration, something happened on the box (low memory?) that caused the wrapper to think that ES was not running and started another instance. The default settings were in place (1 replica, 5 shards), so the new instance took over some of the shards. Upon noticing the immense slowdown on the machine due to two instances running, I killed both process and restarted ES. Of course, the data from the shard moved to the new incorrect instance is missing. I assume that the shards on the original instance were rebalanced, so I cannot simply do a file copy. My plan for data recover is to move the second node data dir to a new location, start a new non-clustered ES server and query from it. Two node clients: one for each server (node). Is it possible to restart two nodes and gracefully tell one of them to accept all merge data? Probably not. -- Ivan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 26, 2011, 2:11pm UTC](https://discuss.elastic.co/t/reindexing-and-error-recovery/4965/2 "2011-07-26T14:11:07Z")
On Tue, Jul 26, 2011 at 3:34 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > Excuse me in advance for the long post. Up until now, I have avoided > rebuilding from source but now is the time to finally dive in. > > Trying to determine reindexing best practices. Using aliases appears to be > the suggested solution to creating updated indices in parallel to searching > an existing index. Searchers always use the same index, while the indexers > create new indexes whenever a reindex is needed, which is then aliased to. > While the new index is being created, do you stop indexing on the existing > index? Scan acts like a cursor, so new inserts after the scan was created > do not get picked up. I think you mean search working against one alias, and then switch the alias from the old index to the new index, right? > Does anybody use a river and reindex data? A river is associated with an > index at creation. If the above recipe holds true, I need to delete the > river and then create a new one with the updated index name. Not the best > scenario since I will lose updates, but the data is not mission critical. River is not really meant for this. You can write simple code that does the reindexing using scan search. In the future, we might have a reindex API, not a river. > On the topic of reindexing, I also need to recover data from an incorrect > node. I switched a single-node elasticsearch server from using monit to > using the service wrapper. On the same day that I made the migration, > something happened on the box (low memory?) that caused the wrapper to think > that ES was not running and started another instance. The default settings > were in place (1 replica, 5 shards), so the new instance took over some of > the shards. Upon noticing the immense slowdown on the machine due to two > instances running, I killed both process and restarted ES. Of course, the > data from the shard moved to the new incorrect instance is missing. I > assume that the shards on the original instance were rebalanced, so I cannot > simply do a file copy. > > My plan for data recover is to move the second node data dir to a new > location, start a new non-clustered ES server and query from it. Two node > clients: one for each server (node). Is it possible to restart two nodes and > gracefully tell one of them to accept all merge data? Probably not. This won't work. Just start the two nodes on the same machine, wait for green health, and then just kill the second one. > -- > Ivan ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [July 26, 2011, 2:40pm UTC](https://discuss.elastic.co/t/reindexing-and-error-recovery/4965/3 "2011-07-26T14:40:47Z")
On Tue, Jul 26, 2011 at 10:11 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > I think you mean search working against one alias, and then switch the > alias from the old index to the new index, right? Correct. A scan will not include new inserts/updates, correct? River is not really meant for this. You can write simple code that does the > reindexing using scan search. In the future, we might have a reindex API, > not a river. Misspoke. My application indexes new content via a river and those indexes need to be updated. I am not suggesting the use of rivers for reindexing. Since the name of the index is specified at river creation time, the river must be deleted and re-added. This won't work. Just start the two nodes on the same machine, wait for > green health, and then just kill the second one. Is that really all? Cannot wait to try it out. I will leave the current directory structure as is and start a new instance via the service wrapper. At this point I am not sure whether I should kill the second instance with the service wrapper (will it knock out both?) or just kill the processes (the second wrapper and instance). Application is still in development mode (pet project, so it probably will always be!). Small, underpowered EC2 instance. Will set replicas to 0 to prevent further issues (after executing the above) while keeping the shards at 5. Might need to start indexing a lot more content, so better machines and a cluster might come in the near future. Currently using a local gateway with ephemeral storage, switching to EBS. -- Ivan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:59am UTC](https://discuss.elastic.co/t/reindexing-and-error-recovery/4965/4 "2017-07-06T03:59:18Z")