# ElasticSearch 1.7: query time spike, and the application processor crash after that **URL:** \ **Category:** Elasticsearch\ **Created:** [November 30, 2017, 2:01am UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673 "2017-11-30T02:01:52Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Weizhi\_Li](https://avatars.discourse-cdn.com/v4/letter/w/2bfe46/32.png) [@Weizhi\_Li](https://discuss.elastic.co/u/Weizhi_Li)\ **Post date:** [November 30, 2017, 2:01am UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/1 "2017-11-30T02:01:53Z")
I used ES as the search engine, the QPS is high (i. e, 1500 rpm), and find some time there is a spike in some nodes, and I have set up timeout is 2s. The traffic is stuck after that. Any ideas to solve this issues? Cache the queries or other solutions? Thanks, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 6:19am UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/2 "2017-11-30T06:19:22Z")
Do you see anything in the logs around that time, e.g. regarding long GC? ---
**Author:** ![Weizhi\_Li](https://avatars.discourse-cdn.com/v4/letter/w/2bfe46/32.png) [@Weizhi\_Li](https://discuss.elastic.co/u/Weizhi_Li)\ **Post date:** [November 30, 2017, 7:29pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/3 "2017-11-30T19:29:47Z")
We did not see anything special in our ES logs, there was no error. We reviewed our cluster metrics at that time and did find the spike in indices.search.query\_time\_in\_millis, but only on one of these data nodes. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 7:32pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/4 "2017-11-30T19:32:09Z")
It doesn't have to be an error. Anything indicating slow merging or long GC could contribute. ---
**Author:** ![Weizhi\_Li](https://avatars.discourse-cdn.com/v4/letter/w/2bfe46/32.png) [@Weizhi\_Li](https://discuss.elastic.co/u/Weizhi_Li)\ **Post date:** [November 30, 2017, 8:07pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/5 "2017-11-30T20:07:21Z")
> [@Christian\_Dahlqvist](#): > > Anything We identified that the merge time (indices.merges.total\_time\_in\_millis) was very high for that particular data node (max 1k for the bad node vs. 100 for the rest). Looks like slow merging was the root cause? Is there a way to solve this issue of node load not being balanced? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 8:30pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/6 "2017-11-30T20:30:04Z")
Look at data distribution. I would also recommend upgrading as you are running a very old version. ---
**Author:** ![Weizhi\_Li](https://avatars.discourse-cdn.com/v4/letter/w/2bfe46/32.png) [@Weizhi\_Li](https://discuss.elastic.co/u/Weizhi_Li)\ **Post date:** [November 30, 2017, 9:21pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/7 "2017-11-30T21:21:12Z")
OK. Why do you think upgrading ES will solve this issue? Do you mean that the new version can handle slow merging or long GC ? For ES 1.7, I am not sure if you can give more suggestions to handle this issue (We reviewed our cluster metrics at that time and did find the spike in indices.search.query\_time\_in\_millis, but only on one of these data nodes.) Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 10:09pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/8 "2017-11-30T22:09:01Z")
What sets the problematic node apart from the others? Are indices being indexed into spread out evenly? Are you using routing or parent-child, which could result in an uneven balance? Do all nodes have the same specification and configuration? ---
**Author:** ![Weizhi\_Li](https://avatars.discourse-cdn.com/v4/letter/w/2bfe46/32.png) [@Weizhi\_Li](https://discuss.elastic.co/u/Weizhi_Li)\ **Post date:** [November 30, 2017, 10:54pm UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/9 "2017-11-30T22:54:33Z")
In our current cluster setup, we have 6 data nodes, while one of them also being master. All nodes have the same specification and configuration despite the node type. We have multiple indices (with various size from 10M to 2G) that can take different amount of traffic. There is no specific routing setup (everything is default). Our indexed documents do not contain parent-child setups. One potential issue we discovered was that, we are using default settings for number of shards / number of replica when building index. Do you have any suggestions on optimal shard / replica assignments in this use case? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 1, 2017, 3:08am UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/10 "2017-12-01T03:08:43Z")
Each nested document is represented by a number of documents behind the scenes. When you update a document all of these are updated. If you have large and/or deep nested documents this can result in a lot of indexing work. How large are your largest nested document? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 29, 2017, 3:09am UTC](https://discuss.elastic.co/t/elasticsearch-1-7-query-time-spike-and-the-application-processor-crash-after-that/109673/11 "2017-12-29T03:09:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.