# Elastic is very slow on bulk insert

**URL:** https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353
**Category:** Elasticsearch
**Created:** [May 5, 2016, 10:29pm UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353 "2016-05-05T22:29:23Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![Pradeep\_Gowda](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@Pradeep\_Gowda](https://discuss.elastic.co/u/Pradeep_Gowda)
#### Post date: [May 5, 2016, 10:29pm UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/1 "2016-05-05T22:29:23Z")

</div>

Hey All,  
I am doing the stress test on Elasticsearch where I'm continuously creating indexes using single thread, maximum of 2000 index. But the index creation time has slowed down as the index count increases, all operations from java client is taking long time to respond and also sometime client is timing out. Does index creation cause any performance issues?

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 6, 2016, 1:06am UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/2 "2016-05-06T01:06:33Z")

</div>

How many shards are you creating per index?

---

<div class="post-metadata">

### Author: ![Pradeep\_Gowda](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@Pradeep\_Gowda](https://discuss.elastic.co/u/Pradeep_Gowda)
#### Post date: [May 6, 2016, 4:08am UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/3 "2016-05-06T04:08:08Z")

</div>

Total 5! As per default settings

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 6, 2016, 4:12am UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/4 "2016-05-06T04:12:27Z")

</div>

That's probably why. 5 shards for 2000 indices = 10000 total indices, way to omany.

---

<div class="post-metadata">

### Author: ![Pradeep\_Gowda](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@Pradeep\_Gowda](https://discuss.elastic.co/u/Pradeep_Gowda)
#### Post date: [May 6, 2016, 4:52am UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/5 "2016-05-06T04:52:04Z")

</div>

So I am collecting millions of record every day and performing aggregation every 30 min on some fields for past 24 hours. In current one index approach, elasitc is becoming too slow. So I am trying horizontal scaling to create multiple index. What is the good approach? One index per hour or day?

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 6, 2016, 5:03am UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/6 "2016-05-06T05:03:55Z")

</div>

Millions a day is not much, why not just use one daily index?

---

<div class="post-metadata">

### Author: ![Pradeep\_Gowda](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@Pradeep\_Gowda](https://discuss.elastic.co/u/Pradeep_Gowda)
#### Post date: [May 6, 2016, 5:11am UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/7 "2016-05-06T05:11:53Z")

</div>

Does it really matter if I create 4 indexes per day or one index per day where there is 500 to 700 million record? Will it improves the performance when there is an aggressive aggregation happen every minute? Our current bottleneck is repeatedly we run into OOM within in one or two days. Even if we give 25GB of a heap to elastic.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 6, 2016, 5:13am UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/8 "2016-05-06T05:13:42Z")

</div>

That's getting out of the scope of your original question, might be better to keep that in the other thread.

---

<div class="post-metadata">

### Author: ![Pradeep\_Gowda](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@Pradeep\_Gowda](https://discuss.elastic.co/u/Pradeep_Gowda)
#### Post date: [May 6, 2016, 5:28am UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/9 "2016-05-06T05:28:57Z")

</div>

Sure. Thank you

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:53pm UTC](https://discuss.elastic.co/t/elastic-is-very-slow-on-bulk-insert/49353/10 "2017-07-05T22:53:25Z")

</div>


