# How to improve performance of Re-indexing from logstash?

**URL:** <https://discuss.elastic.co/t/how-to-improve-performance-of-re-indexing-from-logstash/65966>\
**Category:** Logstash\
**Created:** [November 14, 2016, 11:04am UTC](https://discuss.elastic.co/t/how-to-improve-performance-of-re-indexing-from-logstash/65966 "2016-11-14T11:04:25Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nethis](https://avatars.discourse-cdn.com/v4/letter/n/d2c977/32.png) [@nethis](https://discuss.elastic.co/u/nethis)\
**Post date:** [November 14, 2016, 11:04am UTC](https://discuss.elastic.co/t/how-to-improve-performance-of-re-indexing-from-logstash/65966/1 "2016-11-14T11:04:26Z")

</div>

Hi There,

We have 460 GB of index, which is to be re-index , to make all fields not\_analyzed.

We started re-indexed around 8 days back, and still in progress.(though there are environment issues on shards unavailable exception).

Surprisingly, I could see the new index size as 330GB and then after few minutes, it will reduce to 310 GB. Like this, its been continuing from last 5 days, without increase in the size limit or document count.

Are we missing any configuration here? Please help here

We have  
refresh\_interval : -1  
Replicas:0  
indices.memory.index\_buffer\_size:25%(25% of 56 GB)

logstash configuration as below:

input {  
elasticsearch {  
hosts =\> ["10.158.36.199"]  
index =\> "customevent"  
size =\> 5000  
scroll =\> "20m"  
docinfo =\> true  
}   
}  
output  
{  
elasticsearch {  
action =\> "index"  
hosts =\> ["10.158.36.199"]  
codec =\> json  
index =\> "it\_customevent"  
document\_type =\> "dailyaggregate"  
document\_id =\> "%{[@metadata][\_id]}"   
}  
}

---

<div class="post-metadata">

**Author:** ![eperry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eperry/32/551_2.png) [@eperry](https://discuss.elastic.co/u/eperry)\
**Post date:** [November 14, 2016, 6:42pm UTC](https://discuss.elastic.co/t/how-to-improve-performance-of-re-indexing-from-logstash/65966/2 "2016-11-14T18:42:44Z")

</div>

I don't see anything , anything in the log files? Size may change if your mapping has changed, what is the document count?

FYI, this will never "End" because logstash is supposed to always run, so it will run for the next year if you leave it.

You may want to try just export and re-import the data [https://github.com/taskrabbit/elasticsearch-dump](https://github.com/taskrabbit/elasticsearch-dump) if this is a one time mapping change. There are many other applications, Knapsack is another

---

<div class="post-metadata">

**Author:** ![nethis](https://avatars.discourse-cdn.com/v4/letter/n/d2c977/32.png) [@nethis](https://discuss.elastic.co/u/nethis)\
**Post date:** [November 18, 2016, 8:49am UTC](https://discuss.elastic.co/t/how-to-improve-performance-of-re-indexing-from-logstash/65966/3 "2016-11-18T08:49:27Z")

</div>

Thanks Ed for the suggesting the tools on re-index.

Actually, we just thought of using re-indexing API by re-indexing each day at a time. And this helped me to finish the re-index task in 1-1.5 days(which is around 600 GB).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 16, 2016, 8:49am UTC](https://discuss.elastic.co/t/how-to-improve-performance-of-re-indexing-from-logstash/65966/4 "2016-12-16T08:49:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.