# High latency on search **URL:** \ **Category:** Elasticsearch\ **Created:** [March 11, 2016, 12:10am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092 "2016-03-11T00:10:11Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 11, 2016, 12:10am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/1 "2016-03-11T00:10:11Z")
Hey All, we are using elasticsearch for searching with sort. We perform sorts on 5 fields off which one is a string field which is analyzed. I know its recommended not to sort on analyzed fields but we have to support natural sort on numbers. We have allocated 50% of our 30Gb of heap to fielddata, but when i see the utilization of fielddata it does not go beyond 10GB. and still have very high evictions. We are seeing latencies upto 20Sec for search results . Elasticsearch version - 1.5.2 (in process to upgrade to 1.7) 2.0 is not an option right now. 30 billion documents, with a index size of 24TB 500 shards and 2 replicas. Why is the fielddata not been used entirely and still have high evictions. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2016, 5:44am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/2 "2016-03-11T05:44:38Z")
You potentially have some filters which are cached but never used again. 2.x improves that a lot. How many nodes do you have? ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 11, 2016, 6:31am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/3 "2016-03-11T06:31:27Z")
We have a cluster with 370 data nodes, 150 client nodes and 3 master nodes we index at a rate of 3000 calls per sec and each call will index upto 100 documents we run 3000 queries per second which include just filters, filter with sort and aggs ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 11, 2016, 11:27pm UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/4 "2016-03-11T23:27:42Z")
> [@Arun\_Agarwal](#): > > 30 billion documents, with a index size of 24TB500 shards and 2 replicas. > [@Arun\_Agarwal](#): > > We have a cluster with 370 data nodes, 150 client nodes and 3 master nodeswe index at a rate of 3000 calls per sec and each call will index upto 100 documents > > we run 3000 queries per second That seems like a lot of nodes for that much data and search rate. ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 12, 2016, 12:14am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/5 "2016-03-12T00:14:04Z")
The only reason we added host to the cluster is get more cache for sorting. We have really good performance on filter queries with latency less than 1 sec, but as we add sort to the query performance degrades. Latency for the query spike up to 15-20 secs. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 12, 2016, 12:15am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/6 "2016-03-12T00:15:04Z")
Right, but 370 nodes for 24TB of data is only ~64GB of data per node. That seems highly inefficient. ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 12, 2016, 12:18am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/7 "2016-03-12T00:18:33Z")
What information would be needed to find out why we are getting this high latency and also it would be great if we could reduce the no of nodes. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 12, 2016, 12:38am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/8 "2016-03-12T00:38:53Z")
Are you monitoring ES at all? ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 12, 2016, 12:50am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/9 "2016-03-12T00:50:18Z")
Yes we monitor it. Cluster settings { "persistent" : { "cluster" : { "routing" : { "allocation" : { "allow\_rebalance" : "indices\_all\_active", "node\_concurrent\_recoveries" : "1000", "cluster\_concurrent\_rebalance" : "1", "exclude" : { "\_name" : "" }, "awareness" : { "force" : { "AZ" : { //Awareness based on AZ } } }, "enable" : "all" } } }, "threadpool" : { "refresh" : { "size" : "8", "type" : "scaling" }, "search" : { "queue\_size" : "128", "size" : "32" }, "bulk" : { "size" : "8", "queue\_size" : "128" }, "merge" : { "type" : "scaling", "size" : "8" } }, "indices" : { "cache" : { "filter" : { "size" : "15%" } }, "recovery" : { "concurrent\_streams" : "1000" }, "store" : { "throttle" : { "max\_bytes\_per\_sec" : "150mb" } } } }, "transient" : { "indices" : { "cache" : { "filter" : { "size" : "25%" } } }, "logger" : { "index" : { "search" : { "slowlog" : "DEBUG" } }, "\_root" : "INFO" } } } Cluster status "timestamp" : 1457735896764, "cluster\_name" : "elasticsearch", "uuid" : "Vxv3TygqQVuLq-sSM9luYg", "status" : "green", "indices" : { "count" : 1, "shards" : { "total" : 1500, "primaries" : 500, "replication" : 2.0, "index" : { "shards" : { "min" : 1500, "max" : 1500, "avg" : 1500.0 }, "primaries" : { "min" : 500, "max" : 500, "avg" : 500.0 }, "replication" : { "min" : 2.0, "max" : 2.0, "avg" : 2.0 } } }, "docs" : { "count" : 31462133426, "deleted" : 13086343696 }, "store" : { "size\_in\_bytes" : 28952466994056, "throttle\_time\_in\_millis" : 518656551 }, "fielddata" : { "memory\_size\_in\_bytes" : 3053415346328, "evictions" : 284738711 }, "filter\_cache" : { "memory\_size\_in\_bytes" : 3201933183052, "evictions" : 3369817227 }, "id\_cache" : { "memory\_size\_in\_bytes" : 0 }, "completion" : { "size\_in\_bytes" : 0 }, "segments" : { "count" : 38405, "memory\_in\_bytes" : 284030642242, "index\_writer\_memory\_in\_bytes" : 156242334, "index\_writer\_max\_memory\_in\_bytes" : 804836605952, "version\_map\_memory\_in\_bytes" : 7518960, "fixed\_bit\_set\_memory\_in\_bytes" : 0 }, "percolate" : { "total" : 0, "time\_in\_millis" : 0, "current" : 0, "memory\_size\_in\_bytes" : -1, "memory\_size" : "-1b", "queries" : 0 } }, "nodes" : { "count" : { "total" : 568, "master\_only" : 3, "data\_only" : 410, "master\_data" : 0, "client" : 0 }, "versions" : ["1.5.2"], "os" : { "available\_processors" : 4544, "mem" : { "total\_in\_bytes" : 37401955250176 }, "cpu" : [ { "vendor" : "Intel", "model" : "Xeon", "mhz" : 2500, "total\_cores" : 8, "total\_sockets" : 1, "cores\_per\_socket" : 8, "cache\_size\_in\_bytes" : 25600, "count" : 568 } ] }, "process" : { "cpu" : { "percent" : 83590 }, "open\_file\_descriptors" : { "min" : 15235, "max" : 16081, "avg" : 15627 } }, "jvm" : { "max\_uptime\_in\_millis" : 8719843882, "versions" : [ { "version" : "1.8.0\_51", "vm\_name" : "Java HotSpot(TM) 64-Bit Server VM", "vm\_version" : "25.51-b03", "vm\_vendor" : "Oracle Corporation", "count" : 413 }, { "version" : "1.8.0\_66", "vm\_name" : "Java HotSpot(TM) 64-Bit Server VM", "vm\_version" : "25.66-b17", "vm\_vendor" : "Oracle Corporation", "count" : 155 } ], "mem" : { "heap\_used\_in\_bytes" : 11582235827984, "heap\_max\_in\_bytes" : 17789091840000 }, "threads" : 131409 }, "fs" : { "total\_in\_bytes" : 70323596075008, "free\_in\_bytes" : 39562196209664, "available\_in\_bytes" : 35989967147008, "disk\_reads" : 41684304586, "disk\_writes" : 482283527045, "disk\_io\_op" : 523967831631, "disk\_read\_size\_in\_bytes" : 861052555039744, "disk\_write\_size\_in\_bytes" : 9073652993454080, "disk\_io\_size\_in\_bytes" : 9934705548493824 }, } } ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 12, 2016, 12:52am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/10 "2016-03-12T00:52:06Z")
> [@Arun\_Agarwal](#): > > Yes we monitor it. With what? ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 12, 2016, 12:52am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/11 "2016-03-12T00:52:47Z")
We use r3.2xlarge host, allocated 30Gb of heap to elasticsearch. 50% of the heap to fielddata 25% of the heap to filtercache ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 12, 2016, 12:55am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/12 "2016-03-12T00:55:09Z")
We use the kopf plugin to see cluster status and other information. Do you mean something like marvel ? ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 12, 2016, 12:59am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/13 "2016-03-12T00:59:07Z")
We have a running metrics agent which polls elasticsearch for cluster stats and emit various information from the cluster stats, node stats api. the information collected form these metrics are 1. threadpools 2. cache size (Fitlercache, fielddata) 3. cache evictions 4. segments 5. shard status 6. CPU per host ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 12, 2016, 1:06am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/14 "2016-03-12T01:06:50Z")
So what do they tell you about what is happening on the nodes? Are they overloaded? What does heap/fielddata/fieldcache/threadpool/etc use look like? ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 12, 2016, 1:20am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/15 "2016-03-12T01:20:17Z")
Here is the breakdown of the information 1. heap - 60% average across the cluster, Some host have heap upto 80% utilized 2. fielddata cache - between 5gb to 10Gb across the fleet fielddata evictions - we have one cache eviction every second on an average. 3. filtercache - 7.5Gb across all the host 4. threadpools - max threads 32 for search, we have a surge in the search queue upto 120 but that is only when our traffic spikes 5. threadpool indexing - peak of 8 threads per host, queues upto 256 ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [March 14, 2016, 6:31am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/16 "2016-03-14T06:31:25Z")
Hey one of things i have noticed with the information from our monitoring is that our fielddata cache max usage is 10GB, it does not go beyong 10GB on any host even when i have assigned 50% of the heap to fielddata which should be around 15GB. We have a regular fielddata evictions on our graphs Is there something which is causing this issue ? ---
**Author:** ![Arun\_Agarwal](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@Arun\_Agarwal](https://discuss.elastic.co/u/Arun_Agarwal)\ **Post date:** [April 8, 2016, 2:42am UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/17 "2016-04-08T02:42:49Z")
Removal of sort on analyzed field and using doc\_values for sorting, aggregations reduce the latency by a significant value. Also reduced the number of host. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:01pm UTC](https://discuss.elastic.co/t/high-latency-on-search/44092/18 "2017-07-05T23:01:18Z")