# How to reindex ElasticSearch quickly?

**URL:** https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335
**Category:** Elasticsearch
**Created:** [June 9, 2013, 8:03am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335 "2013-06-09T08:03:25Z")
**Posts on this page:** 15
**Page:** 1

<div class="post-metadata">

### Author: ![Dmitry\_Babitsky](https://avatars.discourse-cdn.com/v4/letter/d/85e7bf/32.png) [@Dmitry\_Babitsky](https://discuss.elastic.co/u/Dmitry_Babitsky)
#### Post date: [June 9, 2013, 8:03am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/1 "2013-06-09T08:03:25Z")

</div>

I have an ElasticSearch index with around 200M documents, total index size  
of 90Gb.

I changed mapping, so I would like ElasticSearch to re-index all the  
documents.

I wrote a script that creates a new index (with the new mapping), then goes  
over all the documents in the old index and puts then into the new one.

It seems to work, but the problem is that it works extremely slowly.  
It started with 300 documents / minute two days ago, and now the speed is  
150 documents/minute.

The script runs on a machine within the same network the elastic search  
machines in.

With such speed it will require a month for the re-index to finish.

Does anybody know about some faster technique to re-index an elastic search  
index?

Thanks in advance!!!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![doug\_livesey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doug_livesey/32/2114_2.png) [@doug\_livesey](https://discuss.elastic.co/u/doug_livesey)
#### Post date: [June 9, 2013, 8:13am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/2 "2013-06-09T08:13:27Z")

</div>

It could be worth looking at the bulk operations -- we rebuild an  
admittedly much smaller index by using the bulk API & loading 2000  
documents in each operation.

On 9 June 2013 09:03, Dmitry Babitsky [dimok21@gmail.com](mailto:dimok21@gmail.com) wrote:

> I have an Elasticsearch index with around 200M documents, total index size  
> of 90Gb.
> 
> I changed mapping, so I would like Elasticsearch to re-index all the  
> documents.
> 
> I wrote a script that creates a new index (with the new mapping), then  
> goes over all the documents in the old index and puts then into the new one.
> 
> It seems to work, but the problem is that it works extremely slowly.  
> It started with 300 documents / minute two days ago, and now the speed is  
> 150 documents/minute.
> 
> The script runs on a machine within the same network the Elasticsearch  
> machines in.
> 
> With such speed it will require a month for the re-index to finish.
> 
> Does anybody know about some faster technique to re-index an elastic  
> search index?
> 
> Thanks in advance!!!
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [June 9, 2013, 8:51am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/3 "2013-06-09T08:51:30Z")

</div>

> **[GitHub - karussell/elasticsearch-reindex: Simple re-indexing. To backup,...](https://github.com/karussell/elasticsearch-reindex)**
>
> Simple re-indexing. To backup, apply index settings changes and more ElasticMagic - GitHub - karussell/elasticsearch-reindex: Simple re-indexing. To backup, apply index settings changes and more El...

Thanks  
Vineeth

On Sun, Jun 9, 2013 at 1:43 PM, doug livesey [biot023@gmail.com](mailto:biot023@gmail.com) wrote:

> It could be worth looking at the bulk operations -- we rebuild an  
> admittedly much smaller index by using the bulk API & loading 2000  
> documents in each operation.
> 
> On 9 June 2013 09:03, Dmitry Babitsky [dimok21@gmail.com](mailto:dimok21@gmail.com) wrote:
> 
> > I have an Elasticsearch index with around 200M documents, total index  
> > size of 90Gb.
> > 
> > I changed mapping, so I would like Elasticsearch to re-index all the  
> > documents.
> > 
> > I wrote a script that creates a new index (with the new mapping), then  
> > goes over all the documents in the old index and puts then into the new one.
> > 
> > It seems to work, but the problem is that it works extremely slowly.  
> > It started with 300 documents / minute two days ago, and now the speed is  
> > 150 documents/minute.
> > 
> > The script runs on a machine within the same network the Elasticsearch  
> > machines in.
> > 
> > With such speed it will require a month for the re-index to finish.
> > 
> > Does anybody know about some faster technique to re-index an elastic  
> > search index?
> > 
> > Thanks in advance!!!
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Dmitry\_Babitsky](https://avatars.discourse-cdn.com/v4/letter/d/85e7bf/32.png) [@Dmitry\_Babitsky](https://discuss.elastic.co/u/Dmitry_Babitsky)
#### Post date: [June 9, 2013, 9:03am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/4 "2013-06-09T09:03:57Z")

</div>

Two questions about the reindex plug-in:

1. Is it possible to reindex an existing index into a new one, so it would  
run offline?
2. I could not understand from the reindex-plug-in readme what is the right  
way to run it, so it will reindex the entire index, without any counters...

Thanks,  
Dmitry.

On Sunday, June 9, 2013 11:51:30 AM UTC+3, Vineeth Mohan wrote:

> [GitHub - karussell/elasticsearch-reindex: Simple re-indexing. To backup, apply index settings changes and more ElasticMagic](https://github.com/karussell/elasticsearch-reindex)
> 
> Thanks  
> Vineeth
> 
> On Sun, Jun 9, 2013 at 1:43 PM, doug livesey \<[bio...@gmail.com](mailto:bio...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > It could be worth looking at the bulk operations -- we rebuild an  
> > admittedly much smaller index by using the bulk API & loading 2000  
> > documents in each operation.
> > 
> > On 9 June 2013 09:03, Dmitry Babitsky \<[dim...@gmail.com](mailto:dim...@gmail.com) \<javascript:\>\>wrote:
> > 
> > > I have an Elasticsearch index with around 200M documents, total index  
> > > size of 90Gb.
> > > 
> > > I changed mapping, so I would like Elasticsearch to re-index all the  
> > > documents.
> > > 
> > > I wrote a script that creates a new index (with the new mapping), then  
> > > goes over all the documents in the old index and puts then into the new one.
> > > 
> > > It seems to work, but the problem is that it works extremely slowly.  
> > > It started with 300 documents / minute two days ago, and now the speed  
> > > is 150 documents/minute.
> > > 
> > > The script runs on a machine within the same network the Elasticsearch  
> > > machines in.
> > > 
> > > With such speed it will require a month for the re-index to finish.
> > > 
> > > Does anybody know about some faster technique to re-index an elastic  
> > > search index?
> > > 
> > > Thanks in advance!!!
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Dmitry\_Babitsky](https://avatars.discourse-cdn.com/v4/letter/d/85e7bf/32.png) [@Dmitry\_Babitsky](https://discuss.elastic.co/u/Dmitry_Babitsky)
#### Post date: [June 9, 2013, 9:40am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/5 "2013-06-09T09:40:43Z")

</div>

The idea of bulk indexing sounds very good!  
One question - how do you perform the bulk _read_?

Thanks a lot!!!

On Sunday, June 9, 2013 11:13:27 AM UTC+3, doug livesey wrote:

> It could be worth looking at the bulk operations -- we rebuild an  
> admittedly much smaller index by using the bulk API & loading 2000  
> documents in each operation.
> 
> On 9 June 2013 09:03, Dmitry Babitsky \<[dim...@gmail.com](mailto:dim...@gmail.com) \<javascript:\>\>wrote:
> 
> > I have an Elasticsearch index with around 200M documents, total index  
> > size of 90Gb.
> > 
> > I changed mapping, so I would like Elasticsearch to re-index all the  
> > documents.
> > 
> > I wrote a script that creates a new index (with the new mapping), then  
> > goes over all the documents in the old index and puts then into the new one.
> > 
> > It seems to work, but the problem is that it works extremely slowly.  
> > It started with 300 documents / minute two days ago, and now the speed is  
> > 150 documents/minute.
> > 
> > The script runs on a machine within the same network the Elasticsearch  
> > machines in.
> > 
> > With such speed it will require a month for the re-index to finish.
> > 
> > Does anybody know about some faster technique to re-index an elastic  
> > search index?
> > 
> > Thanks in advance!!!
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![doug\_livesey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doug_livesey/32/2114_2.png) [@doug\_livesey](https://discuss.elastic.co/u/doug_livesey)
#### Post date: [June 9, 2013, 10:22am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/6 "2013-06-09T10:22:58Z")

</div>

IIRC, you can query for a bunch of documents, and they'll be returned  
(nested in the response) in an array. There must be limit and offset  
options to those queries.  
Once you have have your array of documents, you can feed that to the bulk  
index API.

On 9 June 2013 10:40, Dmitry Babitsky [dimok21@gmail.com](mailto:dimok21@gmail.com) wrote:

> The idea of bulk indexing sounds very good!  
> One question - how do you perform the bulk _read_?
> 
> Thanks a lot!!!
> 
> On Sunday, June 9, 2013 11:13:27 AM UTC+3, doug livesey wrote:
> 
> > It could be worth looking at the bulk operations -- we rebuild an  
> > admittedly much smaller index by using the bulk API & loading 2000  
> > documents in each operation.
> > 
> > On 9 June 2013 09:03, Dmitry Babitsky [dim...@gmail.com](mailto:dim...@gmail.com) wrote:
> > 
> > > I have an Elasticsearch index with around 200M documents, total index  
> > > size of 90Gb.
> > > 
> > > I changed mapping, so I would like Elasticsearch to re-index all the  
> > > documents.
> > > 
> > > I wrote a script that creates a new index (with the new mapping), then  
> > > goes over all the documents in the old index and puts then into the new one.
> > > 
> > > It seems to work, but the problem is that it works extremely slowly.  
> > > It started with 300 documents / minute two days ago, and now the speed  
> > > is 150 documents/minute.
> > > 
> > > The script runs on a machine within the same network the Elasticsearch  
> > > machines in.
> > > 
> > > With such speed it will require a month for the re-index to finish.
> > > 
> > > Does anybody know about some faster technique to re-index an elastic  
> > > search index?
> > > 
> > > Thanks in advance!!!
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > 
> > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > .
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)
#### Post date: [June 9, 2013, 12:35pm UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/7 "2013-06-09T12:35:14Z")

</div>

Yes , you can reindex an existing index.  
You just need to create the new index and give the command to re index it.  
Secondly why it is high speed- It works from within elasticsearch , which  
negates all the network latency when the same thing is done from outside.  
Also it uses scan to "bulk read" and uses bulk insert to copy. So all high  
speed options are used here.

Thanks  
Vineeth

On Sun, Jun 9, 2013 at 2:33 PM, Dmitry Babitsky [dimok21@gmail.com](mailto:dimok21@gmail.com) wrote:

> Two questions about the reindex plug-in:
> 
> 1. Is it possible to reindex an existing index into a new one, so it would  
> run offline?
> 2. I could not understand from the reindex-plug-in readme what is the  
> right way to run it, so it will reindex the entire index, without any  
> counters...
> 
> Thanks,  
> Dmitry.
> 
> On Sunday, June 9, 2013 11:51:30 AM UTC+3, Vineeth Mohan wrote:
> 
> > [https://github.com/karussell/\*\*elasticsearch-reindex](https://github.com/karussell/**elasticsearch-reindex)[https://github.com/karussell/elasticsearch-reindex](https://github.com/karussell/elasticsearch-reindex)
> > 
> > Thanks  
> > Vineeth
> > 
> > On Sun, Jun 9, 2013 at 1:43 PM, doug livesey [bio...@gmail.com](mailto:bio...@gmail.com) wrote:
> > 
> > > It could be worth looking at the bulk operations -- we rebuild an  
> > > admittedly much smaller index by using the bulk API & loading 2000  
> > > documents in each operation.
> > > 
> > > On 9 June 2013 09:03, Dmitry Babitsky [dim...@gmail.com](mailto:dim...@gmail.com) wrote:
> > > 
> > > > I have an Elasticsearch index with around 200M documents, total index  
> > > > size of 90Gb.
> > > > 
> > > > I changed mapping, so I would like Elasticsearch to re-index all the  
> > > > documents.
> > > > 
> > > > I wrote a script that creates a new index (with the new mapping), then  
> > > > goes over all the documents in the old index and puts then into the new one.
> > > > 
> > > > It seems to work, but the problem is that it works extremely slowly.  
> > > > It started with 300 documents / minute two days ago, and now the speed  
> > > > is 150 documents/minute.
> > > > 
> > > > The script runs on a machine within the same network the Elasticsearch  
> > > > machines in.
> > > > 
> > > > With such speed it will require a month for the re-index to finish.
> > > > 
> > > > Does anybody know about some faster technique to re-index an elastic  
> > > > search index?
> > > > 
> > > > Thanks in advance!!!
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > 
> > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > > .
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > 
> > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > .
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Dmitry\_Babitsky](https://avatars.discourse-cdn.com/v4/letter/d/85e7bf/32.png) [@Dmitry\_Babitsky](https://discuss.elastic.co/u/Dmitry_Babitsky)
#### Post date: [June 9, 2013, 1:26pm UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/8 "2013-06-09T13:26:22Z")

</div>

Don't you have long time delays when you put high offsets?  
For example, offset of 800,000 on my DB gives delay of about 30 seconds  
from the time I sent the search command till I start receiving the  
documents.

On Sunday, June 9, 2013 1:22:58 PM UTC+3, doug livesey wrote:

> IIRC, you can query for a bunch of documents, and they'll be returned  
> (nested in the response) in an array. There must be limit and offset  
> options to those queries.  
> Once you have have your array of documents, you can feed that to the bulk  
> index API.
> 
> On 9 June 2013 10:40, Dmitry Babitsky \<[dim...@gmail.com](mailto:dim...@gmail.com) \<javascript:\>\>wrote:
> 
> > The idea of bulk indexing sounds very good!  
> > One question - how do you perform the bulk _read_?
> > 
> > Thanks a lot!!!
> > 
> > On Sunday, June 9, 2013 11:13:27 AM UTC+3, doug livesey wrote:
> > 
> > > It could be worth looking at the bulk operations -- we rebuild an  
> > > admittedly much smaller index by using the bulk API & loading 2000  
> > > documents in each operation.
> > > 
> > > On 9 June 2013 09:03, Dmitry Babitsky [dim...@gmail.com](mailto:dim...@gmail.com) wrote:
> > > 
> > > > I have an Elasticsearch index with around 200M documents, total index  
> > > > size of 90Gb.
> > > > 
> > > > I changed mapping, so I would like Elasticsearch to re-index all the  
> > > > documents.
> > > > 
> > > > I wrote a script that creates a new index (with the new mapping), then  
> > > > goes over all the documents in the old index and puts then into the new one.
> > > > 
> > > > It seems to work, but the problem is that it works extremely slowly.  
> > > > It started with 300 documents / minute two days ago, and now the speed  
> > > > is 150 documents/minute.
> > > > 
> > > > The script runs on a machine within the same network the Elasticsearch  
> > > > machines in.
> > > > 
> > > > With such speed it will require a month for the re-index to finish.
> > > > 
> > > > Does anybody know about some faster technique to re-index an elastic  
> > > > search index?
> > > > 
> > > > Thanks in advance!!!
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > 
> > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > > .
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Dmitry\_Babitsky](https://avatars.discourse-cdn.com/v4/letter/d/85e7bf/32.png) [@Dmitry\_Babitsky](https://discuss.elastic.co/u/Dmitry_Babitsky)
#### Post date: [June 11, 2013, 6:43am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/9 "2013-06-11T06:43:46Z")

</div>

Hi Doug,

I tried your approach, but did not get any time improvement.  
After some debugging I found out that the \*bulk=True \*flag in my index  
command has no effect.

The code that I used is:  
search\_obj = pyes.query.Search(query = pyes.query.MatchAllQuery(), start=  
resume\_from)

old\_index\_iterator = self.esconn.search(search\_obj, self.index\_name)  
counter = 0  
BULK\_SIZE = 2000

for doc in old\_index\_iterator:  
self.esconn._index_(doc=doc, doc\_type=DOC\_TYPE, index=new\_index\_name, id=  
doc.get\_id(),\* bulk=True\*)  
counter += 1

if counter % BULK\_SIZE == 0:  
self.logger.debug("Refreshing...")

- 

```
**self.esconn.refresh()**

```

- 

```
self.logger.debug("Refresh done.")

```

self.esconn.refresh()

Could you please let me know if you use any other pyes API for bulk inserts?

Thanks!!!

On Sunday, June 9, 2013 11:13:27 AM UTC+3, doug livesey wrote:

> It could be worth looking at the bulk operations -- we rebuild an  
> admittedly much smaller index by using the bulk API & loading 2000  
> documents in each operation.
> 
> On 9 June 2013 09:03, Dmitry Babitsky \<[dim...@gmail.com](mailto:dim...@gmail.com) \<javascript:\>\>wrote:
> 
> > I have an Elasticsearch index with around 200M documents, total index  
> > size of 90Gb.
> > 
> > I changed mapping, so I would like Elasticsearch to re-index all the  
> > documents.
> > 
> > I wrote a script that creates a new index (with the new mapping), then  
> > goes over all the documents in the old index and puts then into the new one.
> > 
> > It seems to work, but the problem is that it works extremely slowly.  
> > It started with 300 documents / minute two days ago, and now the speed is  
> > 150 documents/minute.
> > 
> > The script runs on a machine within the same network the Elasticsearch  
> > machines in.
> > 
> > With such speed it will require a month for the re-index to finish.
> > 
> > Does anybody know about some faster technique to re-index an elastic  
> > search index?
> > 
> > Thanks in advance!!!
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)
#### Post date: [June 11, 2013, 8:46am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/10 "2013-06-11T08:46:10Z")

</div>

Hi Dmitry,

You should the use the scan search  
type: [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type/)

In pyes I believe the option is scan=True. Here is a snippet I wrote a  
while ago. Perhaps with an older version than the one you use.

result\_set = es\_client.search(q,indices="index",scan=True,size=batch\_size)

## PATCH pyes for a scanning bug

result\_set.\_max\_item = None

result\_set is now an interable where you can read documents from. pyes will  
make more calls to elasticsearch when needed.

Cheers,  
Boaz

On Tuesday, June 11, 2013 8:43:46 AM UTC+2, Dmitry Babitsky wrote:

> Hi Doug,
> 
> I tried your approach, but did not get any time improvement.  
> After some debugging I found out that the \*bulk=True \*flag in my index  
> command has no effect.
> 
> The code that I used is:  
> search\_obj = pyes.query.Search(query = pyes.query.MatchAllQuery(), start=  
> resume\_from)
> 
> old\_index\_iterator = self.esconn.search(search\_obj, self.index\_name)  
> counter = 0  
> BULK\_SIZE = 2000
> 
> for doc in old\_index\_iterator:  
> self.esconn._index_(doc=doc, doc\_type=DOC\_TYPE, index=new\_index\_name,  
> id=doc.get\_id(),\* bulk=True\*)  
> counter += 1
> 
> if counter % BULK\_SIZE == 0:  
> self.logger.debug("Refreshing...")
> 
> - 
> 
> ```
> **self.esconn.refresh()**
> 
> ```
> 
> - 
> 
> ```
> self.logger.debug("Refresh done.")
> 
> ```
> 
> self.esconn.refresh()
> 
> Could you please let me know if you use any other pyes API for bulk  
> inserts?
> 
> Thanks!!!
> 
> On Sunday, June 9, 2013 11:13:27 AM UTC+3, doug livesey wrote:
> 
> > It could be worth looking at the bulk operations -- we rebuild an  
> > admittedly much smaller index by using the bulk API & loading 2000  
> > documents in each operation.
> > 
> > On 9 June 2013 09:03, Dmitry Babitsky [dim...@gmail.com](mailto:dim...@gmail.com) wrote:
> > 
> > > I have an Elasticsearch index with around 200M documents, total index  
> > > size of 90Gb.
> > > 
> > > I changed mapping, so I would like Elasticsearch to re-index all the  
> > > documents.
> > > 
> > > I wrote a script that creates a new index (with the new mapping), then  
> > > goes over all the documents in the old index and puts then into the new one.
> > > 
> > > It seems to work, but the problem is that it works extremely slowly.  
> > > It started with 300 documents / minute two days ago, and now the speed  
> > > is 150 documents/minute.
> > > 
> > > The script runs on a machine within the same network the Elasticsearch  
> > > machines in.
> > > 
> > > With such speed it will require a month for the re-index to finish.
> > > 
> > > Does anybody know about some faster technique to re-index an elastic  
> > > search index?
> > > 
> > > Thanks in advance!!!
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Dmitry\_Babitsky](https://avatars.discourse-cdn.com/v4/letter/d/85e7bf/32.png) [@Dmitry\_Babitsky](https://discuss.elastic.co/u/Dmitry_Babitsky)
#### Post date: [June 11, 2013, 9:10am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/11 "2013-06-11T09:10:06Z")

</div>

Hi Boaz,

Thanks a lot for your answer.  
According to my measurements, however, the bottleneck is in _index_, which  
ignores \*bulk=True \*flag, not in _search_...

Dmitry.

On Tuesday, June 11, 2013 11:46:10 AM UTC+3, Boaz Leskes wrote:

> Hi Dmitry,
> 
> You should the use the scan search type:  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type/)
> 
> In pyes I believe the option is scan=True. Here is a snippet I wrote a  
> while ago. Perhaps with an older version than the one you use.
> 
> result\_set = es\_client.search(q,indices="index",scan=True,size=batch\_size)
> 
> ## PATCH pyes for a scanning bug
> 
> result\_set.\_max\_item = None
> 
> result\_set is now an interable where you can read documents from. pyes  
> will make more calls to elasticsearch when needed.
> 
> Cheers,  
> Boaz
> 
> On Tuesday, June 11, 2013 8:43:46 AM UTC+2, Dmitry Babitsky wrote:
> 
> > Hi Doug,
> > 
> > I tried your approach, but did not get any time improvement.  
> > After some debugging I found out that the \*bulk=True \*flag in my index  
> > command has no effect.
> > 
> > The code that I used is:  
> > search\_obj = pyes.query.Search(query = pyes.query.MatchAllQuery(), start  
> > =resume\_from)
> > 
> > old\_index\_iterator = self.esconn.search(search\_obj, self.index\_name)  
> > counter = 0  
> > BULK\_SIZE = 2000
> > 
> > for doc in old\_index\_iterator:  
> > self.esconn._index_(doc=doc, doc\_type=DOC\_TYPE, index=new\_index\_name,  
> > id=doc.get\_id(),\* bulk=True\*)  
> > counter += 1
> > 
> > if counter % BULK\_SIZE == 0:  
> > self.logger.debug("Refreshing...")
> > 
> > - 
> > 
> > ```
> > **self.esconn.refresh()**
> > 
> > ```
> > 
> > - 
> > 
> > ```
> > self.logger.debug("Refresh done.")
> > 
> > ```
> > 
> > self.esconn.refresh()
> > 
> > Could you please let me know if you use any other pyes API for bulk  
> > inserts?
> > 
> > Thanks!!!
> > 
> > On Sunday, June 9, 2013 11:13:27 AM UTC+3, doug livesey wrote:
> > 
> > > It could be worth looking at the bulk operations -- we rebuild an  
> > > admittedly much smaller index by using the bulk API & loading 2000  
> > > documents in each operation.
> > > 
> > > On 9 June 2013 09:03, Dmitry Babitsky [dim...@gmail.com](mailto:dim...@gmail.com) wrote:
> > > 
> > > > I have an Elasticsearch index with around 200M documents, total index  
> > > > size of 90Gb.
> > > > 
> > > > I changed mapping, so I would like Elasticsearch to re-index all the  
> > > > documents.
> > > > 
> > > > I wrote a script that creates a new index (with the new mapping), then  
> > > > goes over all the documents in the old index and puts then into the new one.
> > > > 
> > > > It seems to work, but the problem is that it works extremely slowly.  
> > > > It started with 300 documents / minute two days ago, and now the speed  
> > > > is 150 documents/minute.
> > > > 
> > > > The script runs on a machine within the same network the Elasticsearch  
> > > > machines in.
> > > > 
> > > > With such speed it will require a month for the re-index to finish.
> > > > 
> > > > Does anybody know about some faster technique to re-index an elastic  
> > > > search index?
> > > > 
> > > > Thanks in advance!!!
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)
#### Post date: [June 11, 2013, 9:31am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/12 "2013-06-11T09:31:52Z")

</div>

Hi Dmitry,

I'll have to dive into pyes code to see why it goes wrong, but for speed  
you really need to use the bulk api for indexing and the scan search type  
together. If pyes is in your way, you can easily construct the request your  
self using json.dumps (watch out for unicode data and encoding). More info  
here: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/bulk/)

Cheers,  
Boaz

On Tuesday, June 11, 2013 11:10:06 AM UTC+2, Dmitry Babitsky wrote:

> Hi Boaz,
> 
> Thanks a lot for your answer.  
> According to my measurements, however, the bottleneck is in _index_,  
> which ignores \*bulk=True \*flag, not in _search_...
> 
> Dmitry.
> 
> On Tuesday, June 11, 2013 11:46:10 AM UTC+3, Boaz Leskes wrote:
> 
> > Hi Dmitry,
> > 
> > You should the use the scan search type:  
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type/)
> > 
> > In pyes I believe the option is scan=True. Here is a snippet I wrote a  
> > while ago. Perhaps with an older version than the one you use.
> > 
> > result\_set = es\_client.search(q,indices="index",scan=True,size=batch\_size  
> > )
> > 
> > ## PATCH pyes for a scanning bug
> > 
> > result\_set.\_max\_item = None
> > 
> > result\_set is now an interable where you can read documents from. pyes  
> > will make more calls to elasticsearch when needed.
> > 
> > Cheers,  
> > Boaz
> > 
> > On Tuesday, June 11, 2013 8:43:46 AM UTC+2, Dmitry Babitsky wrote:
> > 
> > > Hi Doug,
> > > 
> > > I tried your approach, but did not get any time improvement.  
> > > After some debugging I found out that the \*bulk=True \*flag in my index  
> > > command has no effect.
> > > 
> > > The code that I used is:  
> > > search\_obj = pyes.query.Search(query = pyes.query.MatchAllQuery(),  
> > > start=resume\_from)
> > > 
> > > old\_index\_iterator = self.esconn.search(search\_obj, self.index\_name)  
> > > counter = 0  
> > > BULK\_SIZE = 2000
> > > 
> > > for doc in old\_index\_iterator:  
> > > self.esconn._index_(doc=doc, doc\_type=DOC\_TYPE, index=new\_index\_name,  
> > > id=doc.get\_id(),\* bulk=True\*)  
> > > counter += 1
> > > 
> > > if counter % BULK\_SIZE == 0:  
> > > self.logger.debug("Refreshing...")
> > > 
> > > - 
> > > 
> > > ```
> > > **self.esconn.refresh()**
> > > 
> > > ```
> > > 
> > > - 
> > > 
> > > ```
> > > self.logger.debug("Refresh done.")
> > > 
> > > ```
> > > 
> > > self.esconn.refresh()
> > > 
> > > Could you please let me know if you use any other pyes API for bulk  
> > > inserts?
> > > 
> > > Thanks!!!
> > > 
> > > On Sunday, June 9, 2013 11:13:27 AM UTC+3, doug livesey wrote:
> > > 
> > > > It could be worth looking at the bulk operations -- we rebuild an  
> > > > admittedly much smaller index by using the bulk API & loading 2000  
> > > > documents in each operation.
> > > > 
> > > > On 9 June 2013 09:03, Dmitry Babitsky [dim...@gmail.com](mailto:dim...@gmail.com) wrote:
> > > > 
> > > > > I have an Elasticsearch index with around 200M documents, total index  
> > > > > size of 90Gb.
> > > > > 
> > > > > I changed mapping, so I would like Elasticsearch to re-index all the  
> > > > > documents.
> > > > > 
> > > > > I wrote a script that creates a new index (with the new mapping), then  
> > > > > goes over all the documents in the old index and puts then into the new one.
> > > > > 
> > > > > It seems to work, but the problem is that it works extremely slowly.  
> > > > > It started with 300 documents / minute two days ago, and now the speed  
> > > > > is 150 documents/minute.
> > > > > 
> > > > > The script runs on a machine within the same network the elastic  
> > > > > search machines in.
> > > > > 
> > > > > With such speed it will require a month for the re-index to finish.
> > > > > 
> > > > > Does anybody know about some faster technique to re-index an elastic  
> > > > > search index?
> > > > > 
> > > > > Thanks in advance!!!
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Dmitry\_Babitsky](https://avatars.discourse-cdn.com/v4/letter/d/85e7bf/32.png) [@Dmitry\_Babitsky](https://discuss.elastic.co/u/Dmitry_Babitsky)
#### Post date: [June 20, 2013, 6:47am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/13 "2013-06-20T06:47:47Z")

</div>

Hi Vineeth,

I've installed the \_reindex plug-in, and it works very fast indeed.  
I only have one small problem - I activate the plugin with curl -XPUT  
command from the server Elasticsearch runs from (localhost), and each time  
I run it, it hangs up after several hours.  
The error message that I see coming back from curl is: _curl: (52) Empty  
reply from server_

ubuntu@elasticsearch-test:~$ date; time curl -XPUT  
'[http://localhost:9200/my\_index\_2013\_06\_19\_reindexed/my\_type/\_reindex?searchIndex=my\_index&searchType=my\_type&hitsPerPage=2000](http://localhost:9200/my_index_2013_06_19_reindexed/my_type/_reindex?searchIndex=my_index&searchType=my_type&hitsPerPage=2000)'  
; date  
Wed Jun 19 14:22:20 UTC 2013  
curl: (52) Empty reply from server

real 257m28.136s  
user 0m0.216s  
sys 0m0.460s  
Wed Jun 19 18:39:48 UTC 2013

The example above re-indexed 4M out of 6M of documents that I have.

Any ideas why it goes wrong here?

Thanks,  
Dmitry.

On Sunday, June 9, 2013 11:51:30 AM UTC+3, Vineeth Mohan wrote:

> [GitHub - karussell/elasticsearch-reindex: Simple re-indexing. To backup, apply index settings changes and more ElasticMagic](https://github.com/karussell/elasticsearch-reindex)
> 
> Thanks  
> Vineeth
> 
> On Sun, Jun 9, 2013 at 1:43 PM, doug livesey \<[bio...@gmail.com](mailto:bio...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > It could be worth looking at the bulk operations -- we rebuild an  
> > admittedly much smaller index by using the bulk API & loading 2000  
> > documents in each operation.
> > 
> > On 9 June 2013 09:03, Dmitry Babitsky \<[dim...@gmail.com](mailto:dim...@gmail.com) \<javascript:\>\>wrote:
> > 
> > > I have an Elasticsearch index with around 200M documents, total index  
> > > size of 90Gb.
> > > 
> > > I changed mapping, so I would like Elasticsearch to re-index all the  
> > > documents.
> > > 
> > > I wrote a script that creates a new index (with the new mapping), then  
> > > goes over all the documents in the old index and puts then into the new one.
> > > 
> > > It seems to work, but the problem is that it works extremely slowly.  
> > > It started with 300 documents / minute two days ago, and now the speed  
> > > is 150 documents/minute.
> > > 
> > > The script runs on a machine within the same network the Elasticsearch  
> > > machines in.
> > > 
> > > With such speed it will require a month for the re-index to finish.
> > > 
> > > Does anybody know about some faster technique to re-index an elastic  
> > > search index?
> > > 
> > > Thanks in advance!!!
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)
#### Post date: [June 20, 2013, 9:04am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/14 "2013-06-20T09:04:59Z")

</div>

One more thing that would probably help is disabling the refresh (set  
refresh\_interval to -1) to your new index  
settings[http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html),  
and then change the refresh\_interval back to whatever suits you (1 is the  
default - for auto-refreshing each second).

This also implies that you don't need to do any refresh from your pyes app.  
Btw, in pyes, bulks are automatically sent every bulk\_size (which you have  
to specify when creating the connection object). If you need to flush the  
bulk, there's something like conn.flush() (don't remember the name  
exactly), which does it. You probably want to add that when your script is  
done, although in theory it should flush the bulk on exit.

On Thu, Jun 20, 2013 at 9:47 AM, Dmitry Babitsky [dimok21@gmail.com](mailto:dimok21@gmail.com) wrote:

> Hi Vineeth,
> 
> I've installed the \_reindex plug-in, and it works very fast indeed.  
> I only have one small problem - I activate the plugin with curl -XPUT  
> command from the server Elasticsearch runs from (localhost), and each time  
> I run it, it hangs up after several hours.  
> The error message that I see coming back from curl is: _curl: (52) Empty  
> reply from server_
> 
> ubuntu@elasticsearch-test:~$ date; time curl -XPUT '  
> [http://localhost:9200/my\_index\_2013\_06\_19\_reindexed/my\_type/\_reindex?searchIndex=my\_index&searchType=my\_type&hitsPerPage=2000](http://localhost:9200/my_index_2013_06_19_reindexed/my_type/_reindex?searchIndex=my_index&searchType=my_type&hitsPerPage=2000)  
> '; date  
> Wed Jun 19 14:22:20 UTC 2013  
> curl: (52) Empty reply from server
> 
> real 257m28.136s  
> user 0m0.216s  
> sys 0m0.460s  
> Wed Jun 19 18:39:48 UTC 2013
> 
> The example above re-indexed 4M out of 6M of documents that I have.
> 
> Any ideas why it goes wrong here?
> 
> Thanks,  
> Dmitry.
> 
> On Sunday, June 9, 2013 11:51:30 AM UTC+3, Vineeth Mohan wrote:
> 
> > [https://github.com/karussell/\*\*elasticsearch-reindex](https://github.com/karussell/**elasticsearch-reindex)[https://github.com/karussell/elasticsearch-reindex](https://github.com/karussell/elasticsearch-reindex)
> > 
> > Thanks  
> > Vineeth
> > 
> > On Sun, Jun 9, 2013 at 1:43 PM, doug livesey [bio...@gmail.com](mailto:bio...@gmail.com) wrote:
> > 
> > > It could be worth looking at the bulk operations -- we rebuild an  
> > > admittedly much smaller index by using the bulk API & loading 2000  
> > > documents in each operation.
> > > 
> > > On 9 June 2013 09:03, Dmitry Babitsky [dim...@gmail.com](mailto:dim...@gmail.com) wrote:
> > > 
> > > > I have an Elasticsearch index with around 200M documents, total index  
> > > > size of 90Gb.
> > > > 
> > > > I changed mapping, so I would like Elasticsearch to re-index all the  
> > > > documents.
> > > > 
> > > > I wrote a script that creates a new index (with the new mapping), then  
> > > > goes over all the documents in the old index and puts then into the new one.
> > > > 
> > > > It seems to work, but the problem is that it works extremely slowly.  
> > > > It started with 300 documents / minute two days ago, and now the speed  
> > > > is 150 documents/minute.
> > > > 
> > > > The script runs on a machine within the same network the Elasticsearch  
> > > > machines in.
> > > > 
> > > > With such speed it will require a month for the re-index to finish.
> > > > 
> > > > Does anybody know about some faster technique to re-index an elastic  
> > > > search index?
> > > > 
> > > > Thanks in advance!!!
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > 
> > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > > .
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > 
> > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > .
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:30am UTC](https://discuss.elastic.co/t/how-to-reindex-elasticsearch-quickly/12335/15 "2017-07-06T02:30:14Z")

</div>


