# Poor write performance
**URL:** https://discuss.elastic.co/t/poor-write-performance/153278
**Category:** Elasticsearch
**Created:** [October 21, 2018, 10:08am UTC](https://discuss.elastic.co/t/poor-write-performance/153278 "2018-10-21T10:08:32Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@kluvi](https://discuss.elastic.co/u/kluvi)
#### Post date: [October 21, 2018, 10:08am UTC](https://discuss.elastic.co/t/poor-write-performance/153278/1 "2018-10-21T10:08:32Z")
Hi.
Last few weeks i have performance troubles with write queries (there is bigger load than earlier). I think, that from app side, there is nothing to optimize - all queries ar sent to \_bulk with about 300 operations per request. Most of them are updates and most of them uses update scripts in painless (there is some non-trivial logic). Average duration of one bulk request is about 8-12s, which is terrible. It must be something about 1s (and it earlier was about that time).
All requests goes through RabbitMQ, so I can make some stats about the requests and throttle speed,...
Hardware - we had 3 nodes with enough space, cpu and RAM. (its some virtual servers, but with fast storage) Everything looks good. We tried to add forth node, but without any performance impact. Btw each node has all roles (master, data, ingest,...)
Do you have any suggestions what metrics to watch and how to solve our problem?
I have one idea, but not sure if it can help. I can divide indices into 2 groups, where is almost equal count of writes. Then these indices allocate to different group of nodes (so there will be 2+2 nodes, each containing only one group of indices). Then update the workers (which listens on RabbitMQ queues) to sent requests only to apropriate nodes of cluster.
Splitting to 2 clusters is not possible, because there are also some read requests which needs data from both groups of indices. (read requests are fast enough).
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 21, 2018, 10:51am UTC](https://discuss.elastic.co/t/poor-write-performance/153278/2 "2018-10-21T10:51:40Z")
Which version of Elasticsearch are you using? How frequently are you updating the same documents? Is this something that may have changed as load has increased? Do you have monitoring installed so you can check for trends over time, which may help identify what is going on?
---
### Author:  [@kluvi](https://discuss.elastic.co/u/kluvi)
#### Post date: [October 22, 2018, 7:07am UTC](https://discuss.elastic.co/t/poor-write-performance/153278/3 "2018-10-22T07:07:43Z")
> [@Christian\_Dahlqvist](#):
>
> Which version of Elasticsearch are you using?
6.4.1, JVM: 1.8.0\_181
> [@Christian\_Dahlqvist](#):
>
> How frequently are you updating the same documents?
I havent numbers for this, but it happens from time to time. Some of the bulk requests (not all) have set retry\_on\_conflict=3. My estimate is about 100 conflicts per day
> [@Christian\_Dahlqvist](#):
>
> Is this something that may have changed as load has increased?
I think, that no. Previously, there was no RabbitMQ queue and queries were sent directly from app (which was terrible to debug). Then I rewrite part of app which stores data to separated workers (RabbitMQ), then it works OK about 1 month and then things get wrong...
> [@Christian\_Dahlqvist](#):
>
> Do you have monitoring installed so you can check for trends over time, which may help identify what is going on?
Yes, we use Kibana's monitoring, but there is only 1 week of data. So it is useless to compare "before" and "now"
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 22, 2018, 7:53am UTC](https://discuss.elastic.co/t/poor-write-performance/153278/4 "2018-10-22T07:53:47Z")
If you are updating the same documents frequently, without a refresh having taken place, this can result in a lot of small refreshes that can affect performance negatively. Do you see a lot of very small segments being generated?
---
### Author:  [@kluvi](https://discuss.elastic.co/u/kluvi)
#### Post date: [October 29, 2018, 11:17am UTC](https://discuss.elastic.co/t/poor-write-performance/153278/5 "2018-10-29T11:17:13Z")
I am not sure if I am looking for the correct data... is it in endpoint \_cat/segments?
The output looks like this: [https://drive.google.com/open?id=1btH0w8YkoWq69\_wVCo\_teeTtLs0QPQ51](https://drive.google.com/open?id=1btH0w8YkoWq69_wVCo_teeTtLs0QPQ51) - I have no idea if its too much or not. Our cluster has currently 65 indices and 593 shards and about 3TB of data.
Btw I talked with Philipp Krenn and he assured me, that you are the right person to solve our problem 🙂
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 29, 2018, 11:35am UTC](https://discuss.elastic.co/t/poor-write-performance/153278/6 "2018-10-29T11:35:44Z")
Have you tried capturing and inspecting some of the bulk requests to see if the same documents are repeatedly updated?
---
### Author:  [@kluvi](https://discuss.elastic.co/u/kluvi)
#### Post date: [October 29, 2018, 11:44am UTC](https://discuss.elastic.co/t/poor-write-performance/153278/7 "2018-10-29T11:44:52Z")
I am sure, that in one bulk request, there are no multiple updates of same document. Maybe with different bulk requests in short time, but dont think it can be so offten. But I try to make some check for it and let you know the results.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 29, 2018, 11:46am UTC](https://discuss.elastic.co/t/poor-write-performance/153278/8 "2018-10-29T11:46:19Z")
What does disk usage look like?
---
### Author:  [@kluvi](https://discuss.elastic.co/u/kluvi)
#### Post date: [October 29, 2018, 12:46pm UTC](https://discuss.elastic.co/t/poor-write-performance/153278/9 "2018-10-29T12:46:03Z")
If you mean usage as capacity, then there is enough free space (total space: about 1.2TB per node, free space: 340GB/node)
If you mean speed (taken from `iostat sdX`):
Node: tps kB\_read/s kB\_wrtn/s kB\_read kB\_wrtn
node1 3351,08 74368,52 54220,19 78372430496 57139336538
node2 3015,16 59744,43 51363,28 62937946174 54108797165
node3 3245,06 60662,93 53722,02 63793611777 56494499654
node4 3236,35 72683,20 59176,57 63215987610 51468641728
Which looks OK to me (but I have nothing to compare with). Today we also try to move nodes to 3 physical servers (currently it runs in vmware). There will be also more RAM and CPU threads...
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 29, 2018, 12:46pm UTC](https://discuss.elastic.co/t/poor-write-performance/153278/10 "2018-10-29T12:46:52Z")
What does `iostat -x` give? What kind of storage do you have?
---
### Author:  [@kluvi](https://discuss.elastic.co/u/kluvi)
#### Post date: [October 29, 2018, 12:59pm UTC](https://discuss.elastic.co/t/poor-write-performance/153278/11 "2018-10-29T12:59:04Z")
`iostat -x sdX`
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r\_await w\_await svctm %util
node1 0,04 25,35 3118,88 231,15 74330,23 54222,04 76,75 2,34 0,70 1,24 11,03 0,12 38,74
node2 0,06 21,93 2802,10 213,04 59751,62 51379,14 73,72 3,64 1,21 0,91 5,05 0,11 34,61
node3 0,07 24,26 3022,66 222,85 60677,04 53735,54 70,50 2,00 0,62 1,16 11,59 0,12 39,09
node4 0,43 26,72 2913,43 324,40 72727,95 59190,43 81,49 0,07 0,02 1,46 2,29 0,17 54,48
It is some kind of network storage made of SSD-only drives. Our provider says, that they removed any iops limit on the storage (the new physical servers will have local SSD drives).
---
### Author:  [@kluvi](https://discuss.elastic.co/u/kluvi)
#### Post date: [October 29, 2018, 2:10pm UTC](https://discuss.elastic.co/t/poor-write-performance/153278/12 "2018-10-29T14:10:24Z")
I made some logging and it looks, that there are some "conflicts"... I use redis for this... I scan all bulk requests, that will be send to Elasticsearch, then store ID of the document to Redis with 60s expiration (refresh\_interval=10s) and then during processing the bulk request, check if there already exists the key and if yes, then increase some counter. And the counter still increases, which means, that there are some "conflicts" (=trying to update same document within 60s)... I will dig more deeply into it
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 26, 2018, 2:10pm UTC](https://discuss.elastic.co/t/poor-write-performance/153278/13 "2018-11-26T14:10:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.