# 5.5.0 - JVM Heap and Garbage Collection issues when executing queries **URL:** \ **Category:** Elasticsearch\ **Created:** [October 4, 2018, 10:38pm UTC](https://discuss.elastic.co/t/5-5-0-jvm-heap-and-garbage-collection-issues-when-executing-queries/151127 "2018-10-04T22:38:09Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![awebb](https://avatars.discourse-cdn.com/v4/letter/a/f04885/32.png) [@awebb](https://discuss.elastic.co/u/awebb)\ **Post date:** [October 4, 2018, 10:38pm UTC](https://discuss.elastic.co/t/5-5-0-jvm-heap-and-garbage-collection-issues-when-executing-queries/151127/1 "2018-10-04T22:38:09Z")
Hi, Hoping someone can point me in the right direction with an issue related to JVM heap and garbage collection issues when executing search queries. Here's what I know so far: Use case - Performing queries against an index which is populated with documents (word, excel, pdf, email etc...) - Elastic documents contain properties related to the document, plus a "content" field which contains the extracted content of the document (so, can get quite large depending on the document's content) - Queries are performed ad-hoc by users (currently a very low rate) and are free text searches i.e. find all documents containing certain words or phrases - Index rate for each index is rather low i.e. ~5000 an hour for an example index Hardware/Elastic Setup - VMs running in Azure - 3 master, 3 client and 3 data nodes - Data nodes - 4 vCPU, 14Gb RAM - 7gb RAM allocated to JVM heap, swapping disabled, memory locked Index Configuration - Index per tenant (~30 so far) - 10 primary shards, 1 replica, 30s index refresh - size ranges from 1gb to 400gb (I'm currently testing against a 50Gb index, which is the largest on the specific cluster it is hosted on) - 2 document types, 15 properties, most are keyword, text, Boolean, long, double and one nested property Issue - When performing certain queries (that return a limited number of results i.e. terms are infrequent in the target index), the query will take 15-60+ seconds - In these cases, the data nodes normally log that they are performing garbage collection, which obviously is affecting the query performance - In the slow query log, the fetch phase for certain shards can range from ~200ms to 12+ seconds. Which is reflected by the node GC logging - We have had occasions where OOM exceptions are thrown - Mostly, Kibana shows that there is a massive drop in heap usage due to GC Example - Below is an example query - The "accessControlLevels" terms filter is used to determine security requirements and includes multiple (27 in this case) entries that follow the below patterns "aduser:726877fb-ac7a-48fc-ac90-715a1a558f22" "adgroup:5eff07e9-388f-4219-bd6e-dc746b04fcd2" "email:someone@company.com" `[2018-10-04T20:41:54,177][WARN][index.search.slowlog.fetch] [datanode-2] [fd23a7d3-362d-406b-82ba-7f7155734ab6_1002][2] took[12.7s], took_millis[12783], types[contentobject], stats[], search_type[QUERY_THEN_FETCH], total_shards[10], so urce[{"from":0,"size":25,"query":{"bool":{"must":[{"query_string":{"query":"abc def","fields":[],"use_dis_max":true,"tie_breaker":0.0,"default_operator":"and","auto_generate_phrase_queries":false,"max_determinized_states":10000,"enable_p osition_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"split_on_whitespace":true,"boost":1.0}}],"filter":[{"terms":{"accessControlLevels":[*see explanation*],"boost":1.0}}],"disable_coord":false,"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":["*"],"excludes":["content"]},"sort":[{"uniqueId":{"order":"asc"}}],"highlight":{"requi re_field_match":false,"fields":{"content":{"no_match_size":200}}}}],` - Example node log GC entry below Questions I am hoping to try and answer: 1. I am assuming from the data I have gathered that I need to increase the heap size to reduce the garbage collection. Is this correct? 2. In order to increase the query performance, are there certain caches I should increase, given that the same query may only be run a couple of times to locate a specific document and never again, or not for hours/days 3. Given that it is likely the user issuing the query is looking for terms in the body of the documents, are there any settings I should consider to improve query performance from this perspective? Many thanks in advance for any ideas/suggestions/comments. Adam. ---
**Author:** ![awebb](https://avatars.discourse-cdn.com/v4/letter/a/f04885/32.png) [@awebb](https://discuss.elastic.co/u/awebb)\ **Post date:** [October 11, 2018, 6:54pm UTC](https://discuss.elastic.co/t/5-5-0-jvm-heap-and-garbage-collection-issues-when-executing-queries/151127/2 "2018-10-11T18:54:01Z")
Hi, Is anyone able to provide some pointers on how to gain a deeper understanding of this issue? i.e. what metrics can I leverage to understand the GC behaviour when I am issuing specific queries and how this affects the FETCH phase performance. If more information is needed to provide insight, please let me know. Many thanks, Adam. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 8, 2018, 6:54pm UTC](https://discuss.elastic.co/t/5-5-0-jvm-heap-and-garbage-collection-issues-when-executing-queries/151127/3 "2018-11-08T18:54:02Z")
Closed on November 8, 2018, 6:54pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.