# Elasticsearch bulk update is extremely slow
**URL:** https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258
**Category:** Elasticsearch
**Created:** [March 11, 2017, 8:54pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258 "2017-03-11T20:54:31Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Vignesh\_Chandrasekar](https://discuss.elastic.co/u/Vignesh_Chandrasekar)
#### Post date: [March 11, 2017, 8:54pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/1 "2017-03-11T20:54:31Z")
Hi Team,
I have setup a cluster which indexes ~160GB of data per day into elasticsearch. I am currently facing this case where I need to update almost all the docs in all indices with a small amount of data(~16GB) per index which is of the format
```
id1,data1
id1,data2
id2,data1
id2,data2
id2,data3
.
.
.
```
My update operations start happening at 16000 lines per second and in 5 minutes it comes down to 1000 lines per second and doesnt go up after that. The way it stands now, the time for this update to happen is longer than my entire indexing process for 1 day
My conf file for the update operation currently looks as follows
```
output
{
elasticsearch {
action => "update"
doc_as_upsert => true
hosts => ["host1","host2","host3","host4"]
index => "logstash-2017-08-1"
document_id => "%{uniqueid}"
document_type => "daily"
retry_on_conflict => 2
flush_size => 1000
}
}
```
The optimizations I have done to speed up indexing in my cluster based on the suggestions here [https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html) are
Setting "indices.store.throttle.type" : "none"
Index "refresh\_interval" : "-1"
I am running my cluster on 4 instances of the d2.8xlarge EC2 instances. I have allocated 30GB heap to each node. While the update is happening node cpu is barely used and the load is very less as well.
Is there something very obvious that I am missing that is causing this issue? While looking at the threadpool data I find that the number of threads working on bulk operations are constantly high.
Any help on this issue would be really helpful! Please let me know if you would need more info
Thanks in advance
Vignesh
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 11, 2017, 9:33pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/2 "2017-03-11T21:33:52Z")
What version are you on?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 12, 2017, 1:41am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/3 "2017-03-12T01:41:34Z")
Updates requires Elasticsearch to first find the document and then overwrite it, which tends to get slower the larger the shards gets. If you are reindexing and updating all the data in the index, it may make sense to write the updated data to a new index and then delete the original one when this has completed.
---
### Author:  [@Vignesh\_Chandrasekar](https://discuss.elastic.co/u/Vignesh_Chandrasekar)
#### Post date: [March 12, 2017, 3:31am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/4 "2017-03-12T03:31:09Z")
I am running elasticsearch 5.1 in all the nodes. Each index has 8 shards and 1 replica. So the size of one shard comes to ~20GB
I agree that updates involve first finding the document and then overwriting it which may potentially cause the system to be very slow. However the update initially starts happening at a very fast rate and then slows down tremendously.
I also ran 2 separate logstash instances to simultaneously update different indices. This is causing the write to happen at 500 lps per instance. And my cluster nodes are barely facing any load or using any cpu .
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 12, 2017, 3:46am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/5 "2017-03-12T03:46:40Z")
What does iostat, especially iowait, look like during indexing?
---
### Author:  [@Vignesh\_Chandrasekar](https://discuss.elastic.co/u/Vignesh_Chandrasekar)
#### Post date: [March 12, 2017, 3:55am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/6 "2017-03-12T03:55:58Z")
The update is actually going on right now and this is a snapshot of the iostat for one of the nodes
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 12, 2017, 4:04am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/7 "2017-03-12T04:04:19Z")
That looks quite good. I wonder if Logstash could be the bottleneck? What does the rest of your configuration look like? Which version of Logstash and Elasticsearch are you on?
---
### Author:  [@Vignesh\_Chandrasekar](https://discuss.elastic.co/u/Vignesh_Chandrasekar)
#### Post date: [March 12, 2017, 4:15am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/8 "2017-03-12T04:15:25Z")
I am running logstash 5.2 and Elasticsearch 5.1. I was actually hoping that the bottleneck might be logstash as it is a lot more easier to fix.
But then when I run the exact same logstash configuration with
```
output
{
null{}
}
```
The logstash process works at 36000 lines per second. So it doesnt seem to be the bottleneck here, Or is there something I might be overlooking?
---
### Author:  [@Vignesh\_Chandrasekar](https://discuss.elastic.co/u/Vignesh_Chandrasekar)
#### Post date: [March 12, 2017, 4:19am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/9 "2017-03-12T04:19:48Z")
Also one thing I missed mentioning is that if I run this on an empty index where it upserts documents, the same issue is happening. It runs at almost the same speed. However if it is changed from "update" to "index" the speed picks up and runs at 10000 lines per second
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 12, 2017, 1:17pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/10 "2017-03-12T13:17:12Z")
Have you tried increasing the [the internal batch size within logstash](https://www.elastic.co/guide/en/logstash/current/running-logstash-command-line.html#command-line-flags) (maybe to 1000?) to see if this has an impact? Does performance change if you set the refresh interval to 10s instead of -1?
As I said in my initial response, indexing into a new index may be considerably faster than updating the existing one.
---
### Author:  [@Vignesh\_Chandrasekar](https://discuss.elastic.co/u/Vignesh_Chandrasekar)
#### Post date: [March 13, 2017, 3:49am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/11 "2017-03-13T03:49:31Z")
I've tried all of those and none of it make any noticeable change 😑 played around with everything to the maximum.
I'm actually working on doing that right now. I'm merging my data locally and indexing again. It definitely is much faster than doing the update this way
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 10, 2017, 3:49am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/12 "2017-04-10T03:49:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.