# Slow indexes search **URL:** \ **Category:** Elasticsearch\ **Created:** [June 15, 2019, 8:43am UTC](https://discuss.elastic.co/t/slow-indexes-search/185930 "2019-06-15T08:43:34Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 8:43am UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/1 "2019-06-15T08:43:35Z")
Hi, Each day elasticsearch creates index from two different devices. From the first device I get results really, really slow and even sometimes I got timeout error, from the second one it's still slow but I do not receive any error. They are both on the same machines, and are similar in docs count and size, 1) 107294970 180.8gb, 2) 143319153 202.9gb. The main difference is in fields number 433 vs 204. Could you suggest me what could be wrong, and how to fix this? Maybe I should use more shards per index? Currently there are 2 mains and 2 replicas. Below there are stats from one of the indexes (the same day) per device after commiting search for 7 days. Sorry for the external source but i was unable to paste it all in here. 1) [https://pastebin.com/jdAC0dRu](https://pastebin.com/jdAC0dRu) 1. [https://pastebin.com/Ujme6e5H](https://pastebin.com/Ujme6e5H) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 15, 2019, 9:19am UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/2 "2019-06-15T09:19:13Z")
What kind of searches are you running? What is the indexing and query load? What is the specification of your cluster? What type of storage do you have? ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 1:47pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/3 "2019-06-15T13:47:51Z")
Hi, I think that query is not import, I even get timeout in simple kibana empty query (longer than 7 days). Cluster specifications: all virtual, 8 data node - 20cpu and 60GB ram each Kibana requests query are through coordinator node. None of the cluster member use more than 75% of JVM memory. Elasticsearch version: 6.7.2 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 15, 2019, 2:04pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/4 "2019-06-15T14:04:12Z")
What type of storage are you using? ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 2:09pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/5 "2019-06-15T14:09:02Z")
SSD on 4 first nodes and HDD on the rest, the first ones are keeping 2 days of logs ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 2:12pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/6 "2019-06-15T14:12:27Z")
There are other indexes on this cluster but for some reason, the trouble one is just this one. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 15, 2019, 2:17pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/7 "2019-06-15T14:17:45Z")
What is the result if you query just the last 2 days or day 2 to 7? What does disk IO on the nodes backed by HDD look like while querying? ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 2:20pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/8 "2019-06-15T14:20:15Z")
Ok, why do you think that there is an issue with the IOs? Like I said, there are few similar indexes on this cluster and only one creates issues with searches longer that 7 days. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 15, 2019, 2:22pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/9 "2019-06-15T14:22:25Z")
I am looking to eliminate common causes and disk I/O or iowait is very common. It could also be that the shard for this index is too large for the data you have and the queries you are running. Each query runs single-threaded against each shard so size will affect latency. ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 2:25pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/10 "2019-06-15T14:25:51Z")
So the stats said nothing important in this case? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 15, 2019, 2:28pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/11 "2019-06-15T14:28:03Z")
I could see that you have throttling going on which indicate slow storage. ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 2:31pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/12 "2019-06-15T14:31:01Z")
It's really hard to accept, that there is problem only with this one index.. There are others, bigger in term of doc count and GB. The only real difference is the field number. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 15, 2019, 2:32pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/13 "2019-06-15T14:32:55Z")
You have provided very little information to go on so it is very hard to tell. Is there anything else that sets this index apart from the others, e.g. different types of queries, document size, higher query volume etc? It would also help to know if there is any difference in latency beteen the data store on the different types of nodes. If there is not at least we have ruled that out. ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 2:41pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/14 "2019-06-15T14:41:53Z")
Are there any commands that I could execute for this indices so they could help improve this ticket? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 15, 2019, 3:34pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/15 "2019-06-15T15:34:02Z")
Can you start by comparing query latency against 2 days worth of data on the different node types? Are you running queries or Kibana dashboards? If queries, could you try using the [search profile API](https://www.elastic.co/guide/en/elasticsearch/reference/7.1/search-profile.html)? ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 4:17pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/16 "2019-06-15T16:17:22Z")
Like I said, I even get timeout with simple/empty query 1 index SSD { "took" : 292, "timed\_out" : false, "\_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 101519698, HDD { "took" : 191, "timed\_out" : false, "\_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 111305720, 2 index SSD{ "took" : 410, "timed\_out" : false, "\_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 139194459, HDD{ "took" : 417, "timed\_out" : false, "\_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 165511168, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 15, 2019, 4:20pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/17 "2019-06-15T16:20:55Z")
What does that query look like? Based on that data it seems it executed quite quickly. ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 4:24pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/18 "2019-06-15T16:24:06Z")
It's empty, in this api empty queries doesn't generate any errors, empty (only counting) query does ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 15, 2019, 4:27pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/19 "2019-06-15T16:27:51Z")
Can you show us the exact query? As the number of fields in the index seems to matter this could be explained if you were running a query string query that triggered searching against [all\_fields](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/query-dsl-query-string-query.html#query-dsl-query-string-query). ---
**Author:** ![Necus](https://avatars.discourse-cdn.com/v4/letter/n/e47774/32.png) [@Necus](https://discuss.elastic.co/u/Necus)\ **Post date:** [June 15, 2019, 4:31pm UTC](https://discuss.elastic.co/t/slow-indexes-search/185930/20 "2019-06-15T16:31:24Z")
It's literally empty GET abc-2019.06.\*/\_search no errors and kibana -\> discover -\> index pattern (same time) timeout [Next page](https://discuss.elastic.co/t/slow-indexes-search/185930.md?page=2)