# Random slow query, many futex time **URL:** \ **Category:** Elasticsearch\ **Created:** [January 15, 2019, 2:59am UTC](https://discuss.elastic.co/t/random-slow-query-many-futex-time/164226 "2019-01-15T02:59:13Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![d1fff15e6947696dfc4a](https://avatars.discourse-cdn.com/v4/letter/d/90db22/32.png) [@d1fff15e6947696dfc4a](https://discuss.elastic.co/u/d1fff15e6947696dfc4a)\ **Post date:** [January 15, 2019, 2:59am UTC](https://discuss.elastic.co/t/random-slow-query-many-futex-time/164226/1 "2019-01-15T02:59:14Z")
**System Info** 1. Debian GNU/Linux 8.11 (jessie) 2. ElasticSearch 2.3.5 3. 128G RAM, 1T SSD 4. Elasticsearch 122node, 100 shard， Data size near 100G per shard.（may be too big） **Detail** We use elasticsearch in production, but there are random slow query(P99-\>200ms, P999-\>5000ms). I guess it is relate to cacahe or something? Futher more， **I use strace to show thread info, but find too much time in futex** strace -c -p 11872 % time seconds usecs/call calls errors syscall #------ ----------- ----------- --------- --------- ---------------- 99.78 2.110971 5441 388 55 futex 0.22 0.004741 0 39512 pread 0.00 0.000000 0 35 write #------ ----------- ----------- --------- --------- ---------------- 100.00 2.115712 39935 55 total **Question:** 1. why exist too much futex? It seems ES use mutex rarely 2. desired to get some advise to reduce random slow query Thanks ---
**Author:** ![d1fff15e6947696dfc4a](https://avatars.discourse-cdn.com/v4/letter/d/90db22/32.png) [@d1fff15e6947696dfc4a](https://discuss.elastic.co/u/d1fff15e6947696dfc4a)\ **Post date:** [January 15, 2019, 8:40am UTC](https://discuss.elastic.co/t/random-slow-query-many-futex-time/164226/2 "2019-01-15T08:40:55Z")
**update information** I've discover that query\_cahe has some problem. My index has 1 replica. I've note that primaries query\_cache hit rate about **20%** （44765221451/214657219339）， but total hit rate only **10%** (49212601996/461818150070). **It means replica cache is really low.** What is the reason? Can anyone give me some advice? thanks below is my index stats { "image\_index":{ "primaries":{ "docs":{ "count":2086786254, "deleted":70025697 }, "store":{ "size\_in\_bytes":18458523357144, "throttle\_time\_in\_millis":0 }, "indexing":{ "index\_total":2075625388, "index\_time\_in\_millis":4400528868, "index\_current":1038176, "index\_failed":3466045, "delete\_total":2, "delete\_time\_in\_millis":0, "delete\_current":0, "noop\_update\_total":5428790, "is\_throttled":false, "throttle\_time\_in\_millis":0 }, "search":{ "open\_contexts":281, "query\_total":401787945, "query\_time\_in\_millis":6438147591, "query\_current":148, "fetch\_total":349453018, "fetch\_time\_in\_millis":1100806195, "fetch\_current":1, "scroll\_total":54588, "scroll\_time\_in\_millis":10107052529, "scroll\_current":0 }, "merges":{ "current":2, "current\_docs":24845, "current\_size\_in\_bytes":234600157, "total":371718, "total\_time\_in\_millis":18541380533, "total\_docs":6158738869, "total\_size\_in\_bytes":55118571635584, "total\_stopped\_time\_in\_millis":48077, "total\_throttled\_time\_in\_millis":16850677176, "total\_auto\_throttle\_in\_bytes":524288000 }, "refresh":{ "total":1446338, "total\_time\_in\_millis":304514067 }, "flush":{ "total":36480, "total\_time\_in\_millis":26856479 }, "warmer":{ "current":0, "total":3329153, "total\_time\_in\_millis":199601 }, "query\_cache":{ "memory\_size\_in\_bytes":6019662792, "total\_count":214657219339, "hit\_count":44765221451, "miss\_count":169891997888, "cache\_size":160237, "cache\_count":163216, "evictions":2979 }, "segments":{ "count":6195, "memory\_in\_bytes":12727327254, "terms\_memory\_in\_bytes":10749641354, "stored\_fields\_memory\_in\_bytes":1879112864, "term\_vectors\_memory\_in\_bytes":0, "norms\_memory\_in\_bytes":13196096, "doc\_values\_memory\_in\_bytes":85376940, "index\_writer\_memory\_in\_bytes":118923993, "index\_writer\_max\_memory\_in\_bytes":53687091200, "version\_map\_memory\_in\_bytes":282017, "fixed\_bit\_set\_memory\_in\_bytes":0 } ... }, "total":{ "docs":{ "count":4173572512, "deleted":140174964 }, "store":{ "size\_in\_bytes":36916810920129, "throttle\_time\_in\_millis":0 }, "indexing":{ "index\_total":2148402539, "index\_time\_in\_millis":4585290546, "index\_current":1284700, "index\_failed":5420934, "delete\_total":2, "delete\_time\_in\_millis":0, "delete\_current":0, "noop\_update\_total":5428790, "is\_throttled":false, "throttle\_time\_in\_millis":0 }, "search":{ "open\_contexts":595, "query\_total":803304861, "query\_time\_in\_millis":14959068059, "query\_current":411, "fetch\_total":698641431, "fetch\_time\_in\_millis":2032548499, "fetch\_current":4, "scroll\_total":90797, "scroll\_time\_in\_millis":17631387614, "scroll\_current":0 }, "merges":{ "current":4, "current\_docs":238619, "current\_size\_in\_bytes":2063610536, "total":490320, "total\_time\_in\_millis":19115685770, "total\_docs":6419529805, "total\_size\_in\_bytes":57346589363548, "total\_stopped\_time\_in\_millis":72365, "total\_throttled\_time\_in\_millis":17333190273, "total\_auto\_throttle\_in\_bytes":1048576000 }, "refresh":{ "total":2221358, "total\_time\_in\_millis":341537448 }, "flush":{ "total":47732, "total\_time\_in\_millis":28205924 }, "warmer":{ "current":0, "total":5015007, "total\_time\_in\_millis":301897 }, "query\_cache":{ "memory\_size\_in\_bytes":6530819480, "total\_count":461818150070, "hit\_count":49212601996, "miss\_count":412605548074, "cache\_size":184909, "cache\_count":187904, "evictions":2995 } ... }, ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 12, 2019, 8:41am UTC](https://discuss.elastic.co/t/random-slow-query-many-futex-time/164226/3 "2019-02-12T08:41:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.