# How can I tune for Elasticsearch performance?

**URL:** <https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063>\
**Category:** Elasticsearch\
**Created:** [April 8, 2020, 7:24am UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063 "2020-04-08T07:24:29Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![sgonv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgonv/32/65284_2.png) [@sgonv](https://discuss.elastic.co/u/sgonv)\
**Post date:** [April 8, 2020, 7:24am UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/1 "2020-04-08T07:24:29Z")

</div>

Hello,  
I have the Elasticsearch index with 2 Billion documents.  
And my Elasticsearch cluster has 2 data node with 32GB memory each.

The index looks like below,

```auto
{
	"mappings": {
		"properties": {
  			"userName": {
  				"type": "keyword"
	  		},
	  		"productName": {
	  			"type": "text",
                                "analyzer": "korean_nori_analyzer",
		    	        "fields": {
	  				"standard": {
	  					"type": "text",
	  					"analyzer": "standard"
	  				}
				}
		    	}
                }  
	},
	"settings": {
		"index": {
			"analysis": {
				"tokenizer": {
					"nori_user_dict": {
						"type": "nori_tokenizer",
						"decompound_mode": "mixed"
					}
				},
				"analyzer": {
					"korean_nori_analyzer": {
						"type": "custom",
						"tokenizer": "nori_user_dict"
					}
				}
			},
			"number_of_shards": 10
		}
	}
}

```

And I used 'userName' field for \_routing value. So when I want to find specific user's data, I can find it in only one shard.

Each shard size are 20~30GB, so I think that is no problem.

But when I took GET \_search?routing=user1 operation on my index, it took more than 5 seconds.

```auto
{
	"query": {
		"bool": {
			"must": [
				{
					"multi_match": {
						"query": "삼다수",
						"fields": ["productName", "productName.standard"]
					}
				}
			],
			"filter": [
				{
					"term": {
						"userName": "user1"
					}
				}
			]
		}
	}
}

```

Although I removed query context, query time easily surpassed 5 seconds.

```auto
{
	"query": {
		"bool": {
			"filter": [
				{
					"term": {
						"userName": "user1"
					}
				}
			]
		}
	}
}

```

How can I tune performance for my Elasticsearch index?

I want to get data within 1 seconds.

Is this query time natural for 2 Billion data?

And how about increasing shard number 10 to 100? I guess that the more shard number, the faster performance, because I can query on only one shard.

Thanks.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 8, 2020, 7:34am UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/2 "2020-04-08T07:34:45Z")

</div>

The query latency is often proportional to the shard size so if you have a large number of users a higher primary shard count should allow each query to target less data per routed query and therefore return faster responses. I would also recommend looking at identifying whether there is anything limiting performance, e.g. high disk I/O, iowait, CPU or perhaps long or frequent GC.

How large is the index you are querying? Do you have replica shard enabled? What type of hardware and storage is Elasticsearch deployed on? Which version of Elasticsearch are you using?

---

<div class="post-metadata">

**Author:** ![sgonv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgonv/32/65284_2.png) [@sgonv](https://discuss.elastic.co/u/sgonv)\
**Post date:** [April 8, 2020, 10:58am UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/3 "2020-04-08T10:58:55Z")

</div>

Thanks for the reply.

I increased the shard count into 100, performance was improved.  
But still I have to improve performance.

Is there a limit for shard count?

My index has 2 Billion documents, about 300GB.

There are no striking stat on CPU or disk usage.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 8, 2020, 11:10am UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/4 "2020-04-08T11:10:32Z")

</div>

If you have that much data I would not be surprised if disk I/O and iowait was a limiting factor, especially if you are not using local SSD disks. Whay does `iostat -x -d 2 3` give if you run that while querying?

---

<div class="post-metadata">

**Author:** ![Daniel\_Penning](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_penning/32/48792_2.png) [@Daniel\_Penning](https://discuss.elastic.co/u/Daniel_Penning)\
**Post date:** [April 8, 2020, 12:27pm UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/5 "2020-04-08T12:27:12Z")

</div>

You should also have a look at [index sorting](https://www.elastic.co/guide/en/elasticsearch/reference/master/index-modules-index-sorting.html). Sorting the index by userName can help improve search performance at the cost of a bit slower indexing without involving the overhead of lots of shards.

---

<div class="post-metadata">

**Author:** ![sgonv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgonv/32/65284_2.png) [@sgonv](https://discuss.elastic.co/u/sgonv)\
**Post date:** [April 9, 2020, 4:52am UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/6 "2020-04-09T04:52:07Z")

</div>

I also did experiment with adding index sorting, but performance was not dramatically improved 😢

---

<div class="post-metadata">

**Author:** ![sgonv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sgonv/32/65284_2.png) [@sgonv](https://discuss.elastic.co/u/sgonv)\
**Post date:** [April 9, 2020, 4:55am UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/7 "2020-04-09T04:55:13Z")

</div>

Here is the log for iostat while querying.

```auto
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.01 0.00 0.29 0.00 103.56 0.00 0.48 0.48 0.00 0.10 0.00
sda 0.00 3.79 0.24 70.11 8.10 1077.22 30.85 0.09 1.51 0.44 1.51 0.04 0.28
sdc 0.00 2.35 11.52 141.54 1207.57 8720.53 129.73 0.11 0.73 0.81 0.72 0.09 1.40
sdf 0.00 0.00 0.00 0.00 0.05 0.00 59.00 0.00 0.27 0.27 0.00 0.15 0.00
sde 0.00 0.00 0.00 0.00 0.05 0.00 59.00 0.00 0.27 0.27 0.00 0.15 0.00
sdd 0.00 6.11 117.91 3345.89 14734.56 54020.74 39.70 0.99 0.30 1.60 0.25 0.06 22.16
rbd0 0.00 0.00 0.00 0.00 0.01 0.00 27.04 0.00 1.74 1.74 0.00 1.56 0.00
rbd1 0.00 0.00 0.00 0.00 0.01 0.00 26.70 0.00 4.50 4.50 0.00 4.14 0.00
rbd6 0.00 0.00 0.00 0.00 0.01 0.00 26.58 0.00 4.36 4.36 0.00 3.99 0.00
rbd7 0.03 3.38 15.49 30.95 1524.32 7877.62 404.90 0.45 9.71 7.23 10.96 2.74 12.75
rbd8 0.00 0.00 0.00 0.00 0.02 0.00 33.10 0.00 4.57 4.57 0.00 4.18 0.00
rbd9 0.00 0.00 0.00 0.00 0.04 0.06 35.56 0.00 7.46 4.14 10.31 5.67 0.00
rbd10 0.00 0.03 0.00 3.08 0.07 309.83 201.31 0.02 6.26 5.92 6.26 5.45 1.68
rbd11 0.00 0.00 0.00 0.00 0.01 0.00 28.04 0.00 4.05 4.05 0.00 3.75 0.00
rbd12 0.00 2.31 0.01 8.30 0.25 300.01 72.28 0.05 5.49 12.00 5.49 2.38 1.98
rbd13 0.00 0.00 0.00 0.00 0.01 0.00 27.54 0.00 3.91 3.91 0.00 3.74 0.00
rbd14 0.00 0.00 0.00 0.04 0.02 0.84 46.34 0.00 6.09 9.78 5.99 4.96 0.02
rbd16 0.00 1.70 2.26 16.28 68.91 307.21 40.58 0.12 6.51 18.47 4.85 3.90 7.23
rbd2 0.00 0.00 0.00 0.00 0.01 0.43 155.32 0.00 10.12 6.59 10.64 5.15 0.00
rbd17 0.00 3.24 2.32 83.82 290.51 4540.14 112.15 0.44 5.10 9.81 4.97 2.92 25.16
rbd18 0.00 0.00 0.00 0.02 0.01 0.30 36.73 0.00 4.71 6.81 4.63 3.00 0.00
rbd19 0.00 0.00 0.00 0.00 0.01 0.00 28.20 0.00 1.39 1.39 0.00 1.25 0.00
rbd20 0.00 5.77 0.00 32.31 0.05 1039.40 64.34 0.18 5.45 3.82 5.45 3.71 11.99
rbd21 0.00 0.31 0.08 3.69 6.28 681.29 364.53 0.03 8.59 16.56 8.42 2.59 0.98
rbd5 0.00 0.00 0.00 0.00 0.00 0.04 98.09 0.00 5.26 5.38 5.16 5.43 0.00
rbd22 0.00 0.00 0.00 0.00 0.01 0.00 29.28 0.00 1.60 1.60 0.00 1.40 0.00
rbd23 0.00 0.03 0.00 0.12 0.01 7.38 118.95 0.00 7.11 5.93 7.12 4.22 0.05
rbd3 0.00 0.00 0.00 0.00 0.00 0.00 34.08 0.00 5.28 5.28 5.11 5.22 0.00
rbd4 0.00 0.00 0.00 0.00 0.01 0.00 41.36 0.00 5.56 5.52 32.00 5.55 0.00
rbd24 0.00 0.00 0.00 0.02 0.01 0.97 85.12 0.00 6.72 1.50 7.13 5.66 0.01
rbd15 0.00 0.02 0.11 0.06 5.06 72.15 890.34 0.00 26.51 13.15 50.20 7.59 0.13
rbd25 0.00 0.00 0.00 0.00 0.01 0.00 28.65 0.00 2.20 2.20 0.00 1.78 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.50 0.00 75.00 0.00 1590.00 42.40 0.02 0.43 0.00 0.43 0.03 0.25
sdc 0.00 0.00 0.00 279.00 0.00 16338.00 117.12 0.08 0.31 0.00 0.31 0.05 1.30
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 182.00 4204.00 23200.00 59166.00 37.56 0.69 0.17 1.16 0.13 0.06 26.50
rbd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd7 0.00 8.50 0.00 445.00 0.00 37194.00 167.16 4.04 9.03 0.00 9.03 1.96 87.20
rbd8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd10 0.00 0.00 0.00 3.00 0.00 112.00 74.67 0.04 11.67 0.00 11.67 11.67 3.50
rbd11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd12 0.00 0.00 0.00 3.50 0.00 42.00 24.00 0.02 4.86 0.00 4.86 4.86 1.70
rbd13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd16 0.00 0.00 0.00 27.50 0.00 306.00 22.25 0.21 7.75 0.00 7.75 7.35 20.20
rbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd17 0.00 6.00 205.50 99.00 26304.00 10036.00 238.69 4.86 15.73 18.80 9.35 3.28 100.00
rbd18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd20 0.00 7.00 0.00 55.50 0.00 708.00 25.51 0.29 5.29 0.00 5.29 4.31 23.90
rbd21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 0.00 0.00 0.00 60.00 0.00 3336.00 111.20 0.04 0.76 0.00 0.76 0.04 0.25
sdc 0.00 0.00 0.00 520.00 0.00 15686.00 60.33 0.07 0.15 0.00 0.15 0.05 2.55
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 7.50 0.00 4418.00 0.00 100142.00 45.33 1.13 0.27 0.00 0.27 0.04 19.85
rbd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd7 0.00 22.50 0.00 653.00 0.00 31260.00 95.74 7.14 10.80 0.00 10.80 1.47 96.10
rbd8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd10 0.00 0.00 0.00 3.50 0.00 154.00 88.00 0.01 3.14 0.00 3.14 3.00 1.05
rbd11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd12 0.00 1.00 0.00 2.50 0.00 34.00 27.20 0.01 3.80 0.00 3.80 3.60 0.90
rbd13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd16 0.00 2.00 0.00 20.50 0.00 334.00 32.59 0.19 9.27 0.00 9.27 5.80 11.90
rbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd17 0.00 1.00 207.00 75.00 26386.00 3192.00 209.77 4.39 16.04 18.21 10.04 3.50 98.75
rbd18 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd19 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd20 0.00 9.00 0.00 69.00 0.00 898.00 26.03 0.40 5.78 0.00 5.78 4.17 28.80
rbd21 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
rbd25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 9, 2020, 5:42am UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/8 "2020-04-09T05:42:56Z")

</div>

You have done very high utilisation in there so it seems you are likely limited by the performance of your storage as it is very unevenly distributed. What types of disks do you have? How have you configured them? Are you indexing at the same time you are querying?

---

<div class="post-metadata">

**Author:** ![Jeff\_Bolle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeff_bolle/32/52060_2.png) [@Jeff\_Bolle](https://discuss.elastic.co/u/Jeff_Bolle)\
**Post date:** [April 20, 2020, 2:35pm UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/9 "2020-04-20T14:35:00Z")

</div>

We have some indexes with similar numbers of documents, though our documents are much larger than yours and we have them spread out across a much larger number of nodes. Given the IO stats you previously showed, I think you may benefit from adding more smaller nodes to the cluster with very fast local storage (we use local NVMe disks on each node). This has proven to be a high performance solution for us.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 18, 2020, 2:35pm UTC](https://discuss.elastic.co/t/how-can-i-tune-for-elasticsearch-performance/227063/10 "2020-05-18T14:35:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.