# Elasticsearch 2.3 poor performance
**URL:** https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359
**Category:** Elasticsearch
**Created:** [August 5, 2016, 3:13pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359 "2016-08-05T15:13:00Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Bit](https://discuss.elastic.co/u/Bit)
#### Post date: [August 5, 2016, 3:13pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/1 "2016-08-05T15:13:00Z")
I have ES 1.7.5 cluster with 20 servers. Maximum indexing performance of this cluster is about 6000-7000 docs/sec. Average size of document is about 1.5 Kbytes
First (optional 🙂 question what do you think about this speed?
And for upgrading to ES 2.3 I setup cluster with another 20 servers and run ES 2.3 but maximum performance of this cluster is about 3000-3500 docs/sec. How to improve speed?
Configurations of clusters are almost default except all strings are not\_analyzed. And any server may become master.
Any ideas please.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 5, 2016, 4:02pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/2 "2016-08-05T16:02:53Z")
What type of hardware are you using? What is the specification of the nodes? How are you ingesting data? What does load on the servers look like while you are indexing? How many indices/shards are you actively indexing into?
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [August 5, 2016, 4:08pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/3 "2016-08-05T16:08:18Z")
> [@Bit](#):
>
> Maximum indexing performance of this cluster is about 6000-7000 docs/sec
Can you specifiy this in MB/sec?
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [August 5, 2016, 4:18pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/4 "2016-08-05T16:18:02Z")
Sorry - I see now 1.5 KB avg size. It means 9-10 MB/sec. This is very bad performance for 20 servers, it would mean one server could take only 500 KB/sec.
---
### Author:  [@tinle](https://discuss.elastic.co/u/tinle)
#### Post date: [August 5, 2016, 4:20pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/5 "2016-08-05T16:20:18Z")
This is probably the cause. ES 2.x default for translog changed from async to sync. If you set the following in your ES 2.3 cluster, do you get better performance?
```
index.translog.durability: async
```
---
### Author:  [@jasontedor](https://discuss.elastic.co/u/jasontedor)
#### Post date: [August 5, 2016, 4:24pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/6 "2016-08-05T16:24:47Z")
Let's take caution here. First, when making any change to the `index.translog.durability`, it's immensely important to point out that the tradeoff is a loss of safety. Second, the performance reported here is so incredibly low that I'm _skeptical_ that the best solution to getting back some of the performance is by adjusting the translog sync. I suspect that performance is being left on the table somewhere else and we should focus on understanding that.
---
### Author:  [@tinle](https://discuss.elastic.co/u/tinle)
#### Post date: [August 8, 2016, 4:19pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/7 "2016-08-08T16:19:18Z")
It depends on the use case. For our use case, we're fine with taking that risk.
Agreed that the numbers reported is poor.
---
### Author:  [@Bit](https://discuss.elastic.co/u/Bit)
#### Post date: [August 8, 2016, 4:27pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/8 "2016-08-08T16:27:54Z")
> [@tinle](#):
>
> index.translog.durability: async
Cluster ES 2.3.3 with this option process 12000-18000 docs/sec
Much better then ES 1.7.5 with 6000-7000 docs/sec
I understand risks of use this option.
---
### Author:  [@jasontedor](https://discuss.elastic.co/u/jasontedor)
#### Post date: [August 8, 2016, 4:29pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/9 "2016-08-08T16:29:20Z")
> [@tinle](#):
>
> It depends on the use case.
Yes, which is why it's important when you recommend someone turning off translog durability that you make them aware of the safety tradeoffs.
---
### Author:  [@Bit](https://discuss.elastic.co/u/Bit)
#### Post date: [August 8, 2016, 4:50pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/10 "2016-08-08T16:50:38Z")
Nodes spec:
20 servers per cluster
OS: Linux
CPU: Core i7-6700 CPU @ 3.40GHz
RAM: 64Gb (but 6 nodes in both clusters still has 48Gb - they waits upgrade)
SSD: no
HDD: software raid0 2x2Tb
ES Heap size: 31Gb
Index:
Cluster contain 2 daily rotate indices (index1-YYYY.MM.DD and index2-YYYY.MM.DD)
Both indices has almost the same size.
5 shards per index with 1 replica (2 copies of data)
every daily index contain about 160,000,000-220,000,000 docs
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [August 8, 2016, 5:44pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/11 "2016-08-08T17:44:28Z")
> [@Bit](#):
>
> HDD: software raid0 2x2Tb
Do not use software RAID. Most important, poor disk I/O solutions will thwart powerful CPU like i7-6700. Also, software RAID0, plus disabled transaction durability, is an invitation to data loss. You should use hardware RAID with an optimized file system setting for maximum throughput. Also check if the 2TB drives are built for server performance tasks, or for archival purpose.
You can double indexing speed by the following procedure: 1) create new index with replica level 0 2) bulk index 3) add replica level 1 before enabling search on that index in application.
But I think that alone does still not explain the poor performance.
---
### Author:  [@unknownunknown](https://discuss.elastic.co/u/unknownunknown)
#### Post date: [August 9, 2016, 12:56pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/12 "2016-08-09T12:56:41Z")
> [@jprante](#):
>
> You should use hardware RAID with an optimized file system setting for maximum throughput.
Could you clarify this? Are you talking about matching stripe sizes, etc?
---
### Author:  [@Borrelworst](https://discuss.elastic.co/u/Borrelworst)
#### Post date: [August 9, 2016, 2:44pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/13 "2016-08-09T14:44:53Z")
How is your cluster being setup: Do you have Data nodes only or you also use a Master node? I have seen great performance when I started using a Master node next to the Data nodes.
Also, how much memory do you allocate to the Elasticsearch instances, so what is the value you use for ES\_HEAP\_SIZE ?
---
### Author:  [@Bit](https://discuss.elastic.co/u/Bit)
#### Post date: [August 9, 2016, 2:58pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/14 "2016-08-09T14:58:01Z")
I can not use HW raid.
I can use software raid level 0 or level 1 of two disks or use separately mounted two disks per server.
---
### Author:  [@Bit](https://discuss.elastic.co/u/Bit)
#### Post date: [August 9, 2016, 2:58pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/15 "2016-08-09T14:58:58Z")
I use data nodes only. Any node may become master. ES\_HEAP\_SIZE=31G
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 9, 2016, 3:02pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/16 "2016-08-09T15:02:12Z")
Are you using bulk indexing? If so, what bulk size do you use?
---
### Author:  [@Bit](https://discuss.elastic.co/u/Bit)
#### Post date: [August 9, 2016, 3:07pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/17 "2016-08-09T15:07:42Z")
logstash output config:
> elasticsearch {
> hosts =\> ["XXX.XXX.XXX.XXX:9200"]
> index =\> "common-%{+YYYY.MM.dd}"
> timeout =\> 40
> flush\_size =\> 2000
> }
---
### Author:  [@Borrelworst](https://discuss.elastic.co/u/Borrelworst)
#### Post date: [August 9, 2016, 3:30pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/18 "2016-08-09T15:30:10Z")
Can you try to setup it up with one Masternode which contain no data ? It can also improve the performance on your ES 1.7 cluster.
---
### Author:  [@Damian\_Pfister](https://discuss.elastic.co/u/Damian_Pfister)
#### Post date: [August 11, 2016, 4:05pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/19 "2016-08-11T16:05:11Z")
Any suggestion that ES (either version) is hitting any threadpool limits and dropping events (e.g. bulk rejections)?
What about logstash - any suggestion in the logs that it is not able to keep up?
The [cat API gives great insight](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-thread-pool.html) into this.
Do you have any monitoring solution to give metrics on performance (Marvel, ElasticHQ, elasticsearch-head)?
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [August 11, 2016, 6:58pm UTC](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359/20 "2016-08-11T18:58:43Z")
> [@unknownunknown](#):
>
> Could you clarify this? Are you talking about matching stripe sizes, etc?
Avoid RAID5/6, prefer RAID 0/1/1+0, enlarge read ahead settings with RAID 1+0, match stripe size of file system creation with controller settings of read ahead blocks, add mount options (for XFS nobarrier,noatime,nodiratime), tune kernel I/O scheduler/elevator for high IOPS (maybe queue or even noop alows more throughput than deadline)
Most important: run your benchmarks to be sure to find optimal settings.
[Next page](https://discuss.elastic.co/t/elasticsearch-2-3-poor-performance/57359.md?page=2)