# Term filter causes memory to spike drastically **URL:** \ **Category:** Elasticsearch\ **Created:** [November 21, 2014, 12:07pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887 "2014-11-21T12:07:57Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Ajay\_Divakaran](https://avatars.discourse-cdn.com/v4/letter/a/779978/32.png) [@Ajay\_Divakaran](https://discuss.elastic.co/u/Ajay_Divakaran)\ **Post date:** [November 21, 2014, 12:07pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/1 "2014-11-21T12:07:57Z")
The term filter that is used: curl -XGET '[http://localhost:9200/my-index/my-doc-type/\_search](http://localhost:9200/my-index/my-doc-type/_search)' -d '{ "filter": { "term": {"void": false} }, "fields": [["user\_id1", "user\_name", "date", "status", "q1", "q1\_unique\_code", "q2", "q3"]], "size": 50000, "sort": ["date\_value"]}' - The 'void' field is a boolean field. - The index store size is 504mb. - The elastic search setup consists of only a single node and the index consists of only a single shard and 0 replicas. The version of elasticsearch is 0.90.7 - The fields mentioned above is only the first 8 fields. The actual term filter that we execute has 350 fields mentioned. _We noticed the memory spiking by about 2-3gb though the store size is only 504mb._ _Running the query multiple times seems to continuously increase the memory._ Could someone explain why this memory spike occurs? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nickcanz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickcanz/32/44873_2.png) [@nickcanz](https://discuss.elastic.co/u/nickcanz)\ **Post date:** [November 21, 2014, 12:44pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/2 "2014-11-21T12:44:57Z")
This is something that I just "discovered" as well. Using a top-level filter is really a "post\_filter" (it's renamed in later versions of ES): > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. So, this will execute the query first (a default "match\_all: {}") and then execute the filter on that result set. This is not very efficient for your query, since I expect you expected having filter there to work like a "pre-filter" and filter out results _before_ executing the query. To do that, you need to use a "filtered query": > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. In your case, the resulting query would look like: curl -XGET '[http://localhost:9200/my-index/my-doc-type/\_search](http://localhost:9200/my-index/my-doc-type/_search)' -d '{ "query": { "filtered": { "filter": { "term": {"void": false} } } }, "fields": [["user\_id1", "user\_name", "date", "status", "q1", "q1\_unique\_code", "q2", "q3"]], "size": 50000, "sort": ["date\_value"]}' On Fri, Nov 21, 2014 at 7:07 AM, Ajay Divakaran [ajay.divakaran86@gmail.com](mailto:ajay.divakaran86@gmail.com) wrote: > The term filter that is used: > > curl -XGET '[http://localhost:9200/my-index/my-doc-type/\_search](http://localhost:9200/my-index/my-doc-type/_search)' -d '{ > "filter": { > "term": {"void": false} > }, > "fields": [["user\_id1", "user\_name", "date", "status", "q1", > "q1\_unique\_code", "q2", "q3"]], > "size": 50000, "sort": ["date\_value"]}' > > - The 'void' field is a boolean field. > - The index store size is 504mb. > - The Elasticsearch setup consists of only a single node and the > index consists of only a single shard and 0 replicas. The version of > elasticsearch is 0.90.7 > - The fields mentioned above is only the first 8 fields. The actual > term filter that we execute has 350 fields mentioned. > > _We noticed the memory spiking by about 2-3gb though the store size is > only 504mb._ > > _Running the query multiple times seems to continuously increase the > memory._ > > Could someone explain why this memory spike occurs? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Nick Canzoneri Developer, Wildbit [http://wildbit.com/](http://wildbit.com/) Beanstalk [http://beanstalkapp.com/](http://beanstalkapp.com/), Postmark [http://postmarkapp.com/](http://postmarkapp.com/), > **[DeployBot | Code Deployment Tools | Deploy Code Anywhere](https://deploybot.com)** > > Push. Build. Deploy! Instantly build and ship code anywhere in one consistent process for your entire team. DeployBot's code deployment tools work with your existing git repository to deploy new code fast, and with zero downtime. These are the... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKWm5yPQo5G0w2ShxtC4E9K90Bji1tQca3RqcG%2BvGUDR8aAqMQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKWm5yPQo5G0w2ShxtC4E9K90Bji1tQca3RqcG%2BvGUDR8aAqMQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ajay\_Divakaran](https://avatars.discourse-cdn.com/v4/letter/a/779978/32.png) [@Ajay\_Divakaran](https://discuss.elastic.co/u/Ajay_Divakaran)\ **Post date:** [November 21, 2014, 2:58pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/3 "2014-11-21T14:58:05Z")
So does that mean all the documents in the shard (since the index has only 1 shard) are pulled into memory and then the filter is applied? By moving it to a post-filter, I see the response coming back in around 15s than previously where it was taking more than a minute. However the memory still increases around 2-3GB. Re-running the filtered query again multiple times does not further increase the memory. Though the index store mentions only 504mb could you explain why the memory spikes to 2-3GB even with a filtered-query? With the filtered query approach does the filtering happen at the disk level? Could you also explain why I don't see the memory increasing further with multiple runs of the filtered-query? On Friday, November 21, 2014 6:15:27 PM UTC+5:30, Nick Canzoneri wrote: > This is something that I just "discovered" as well. > > Using a top-level filter is really a "post\_filter" (it's renamed in later > versions of ES): > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-post-filter.html) > > So, this will execute the query first (a default "match\_all: {}") and then > execute the filter on that result set. This is not very efficient for your > query, since I expect you expected having filter there to work like a > "pre-filter" and filter out results _before_ executing the query. > > To do that, you need to use a "filtered query": > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-filtered-query.html) > > In your case, the resulting query would look like: > > curl -XGET '[http://localhost:9200/my-index/my-doc-type/\_search](http://localhost:9200/my-index/my-doc-type/_search)' -d '{ > "query": { > "filtered": { > "filter": { > "term": {"void": false} > } > } > }, > "fields": [["user\_id1", "user\_name", "date", "status", "q1", > "q1\_unique\_code", "q2", "q3"]], > "size": 50000, "sort": ["date\_value"]}' > > On Fri, Nov 21, 2014 at 7:07 AM, Ajay Divakaran \<[ajay.div...@gmail.com](mailto:ajay.div...@gmail.com) > \\> wrote: > > > The term filter that is used: > > > > curl -XGET '[http://localhost:9200/my-index/my-doc-type/\_search](http://localhost:9200/my-index/my-doc-type/_search)' -d '{ > > "filter": { > > "term": {"void": false} > > }, > > "fields": [["user\_id1", "user\_name", "date", "status", "q1", > > "q1\_unique\_code", "q2", "q3"]], > > "size": 50000, "sort": ["date\_value"]}' > > > > - The 'void' field is a boolean field. > > - The index store size is 504mb. > > - The Elasticsearch setup consists of only a single node and the > > index consists of only a single shard and 0 replicas. The version of > > elasticsearch is 0.90.7 > > - The fields mentioned above is only the first 8 fields. The actual > > term filter that we execute has 350 fields mentioned. > > > > _We noticed the memory spiking by about 2-3gb though the store size is > > only 504mb._ > > > > _Running the query multiple times seems to continuously increase the > > memory._ > > > > Could someone explain why this memory spike occurs? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Nick Canzoneri > Developer, Wildbit [http://wildbit.com/](http://wildbit.com/) > Beanstalk [http://beanstalkapp.com/](http://beanstalkapp.com/), Postmark [http://postmarkapp.com/](http://postmarkapp.com/), > [dploy.io](http://dploy.io) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d6436153-49b1-4a07-ab57-d135e035f84d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d6436153-49b1-4a07-ab57-d135e035f84d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [November 21, 2014, 3:14pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/4 "2014-11-21T15:14:57Z")
Hi Ajay, As Nick pointed out, this query is going to match documents on a doc-by-doc basis, which is going to be very slow. However, each iteration is not supposed to increase memory usage. Memory usage might jump on the first request because elasticsearch will need to load the `date_value` field into field data and potentially your term filter into the filter cache, but that should be it, and subsequent executions of this request should not add 2GB of garbage. Something that is uncommonly high in your query is the `size` parameter. Do you have an idea of how large your documents are? It could be that part of the reason why so much garbage is generated is due to the building and then serialization of the search response. On Fri, Nov 21, 2014 at 1:07 PM, Ajay Divakaran [ajay.divakaran86@gmail.com](mailto:ajay.divakaran86@gmail.com) wrote: > The term filter that is used: > > curl -XGET '[http://localhost:9200/my-index/my-doc-type/\_search](http://localhost:9200/my-index/my-doc-type/_search)' -d '{ > "filter": { > "term": {"void": false} > }, > "fields": [["user\_id1", "user\_name", "date", "status", "q1", > "q1\_unique\_code", "q2", "q3"]], > "size": 50000, "sort": ["date\_value"]}' > > - The 'void' field is a boolean field. > - The index store size is 504mb. > - The Elasticsearch setup consists of only a single node and the > index consists of only a single shard and 0 replicas. The version of > elasticsearch is 0.90.7 > - The fields mentioned above is only the first 8 fields. The actual > term filter that we execute has 350 fields mentioned. > > _We noticed the memory spiking by about 2-3gb though the store size is > only 504mb._ > > _Running the query multiple times seems to continuously increase the > memory._ > > Could someone explain why this memory spike occurs? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5pdoVmgSp-27pWQvzJ-aHWqbKDAubmuMN9RUVWCvndDw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5pdoVmgSp-27pWQvzJ-aHWqbKDAubmuMN9RUVWCvndDw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ajay\_Divakaran](https://avatars.discourse-cdn.com/v4/letter/a/779978/32.png) [@Ajay\_Divakaran](https://discuss.elastic.co/u/Ajay_Divakaran)\ **Post date:** [November 21, 2014, 3:37pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/5 "2014-11-21T15:37:29Z")
This query is used for exporting data from our application hence the high value of the size parameter. The doc by doc comparison is done by a term filter or filtered query? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5d12257b-9b9b-4ec6-bfae-f105fa0a5892%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d12257b-9b9b-4ec6-bfae-f105fa0a5892%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ajay\_Divakaran](https://avatars.discourse-cdn.com/v4/letter/a/779978/32.png) [@Ajay\_Divakaran](https://discuss.elastic.co/u/Ajay_Divakaran)\ **Post date:** [November 21, 2014, 3:43pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/6 "2014-11-21T15:43:31Z")
Each document has around 350 text fields. But im still not able to relate the store size to the memory spike. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c8014753-b0e2-481c-85dc-8a9eceae7d32%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c8014753-b0e2-481c-85dc-8a9eceae7d32%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [November 21, 2014, 5:27pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/7 "2014-11-21T17:27:02Z")
One difference is that the store is compressed while the structure of the response that is built into memory is not (and potentially very wasteful). The doc-by-doc comparison is done by the post-filter: for every document that matches the query, the filter is evaluated in order to know whether it matches or not. On the other hand, when a filter is in the query (either under a constant\_score or a filtered\_query), it can efficiently jump to the next matches using the inverted index. If you want to export your index, a more efficient way would be to use the `scan` search type: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html). It basically opens a cursor that you can iterate on, as opposed as trying to get everything at once. On Fri, Nov 21, 2014 at 4:43 PM, Ajay Divakaran [ajay.divakaran86@gmail.com](mailto:ajay.divakaran86@gmail.com) wrote: > Each document has around 350 text fields. > > But im still not able to relate the store size to the memory spike. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/c8014753-b0e2-481c-85dc-8a9eceae7d32%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c8014753-b0e2-481c-85dc-8a9eceae7d32%40googlegroups.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6jsKSLZFxkGgmw%2BQbmwr8WYyeWh\_zqZWQOdX7Y4kivWA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6jsKSLZFxkGgmw%2BQbmwr8WYyeWh_zqZWQOdX7Y4kivWA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ajay\_Divakaran](https://avatars.discourse-cdn.com/v4/letter/a/779978/32.png) [@Ajay\_Divakaran](https://discuss.elastic.co/u/Ajay_Divakaran)\ **Post date:** [November 21, 2014, 5:50pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/8 "2014-11-21T17:50:28Z")
Thanks Adrien for your reply. I upgraded to ES 0.90.13. What I'm now noticing is that the memory seems to continuously increase when running the query again and again. I also noticed that after upgrading it started using OpenJDK 1.6\_0.33. So I switched back to using Oracle JDK 1.7.71 however the issue seems to persist. On Friday, November 21, 2014 10:57:09 PM UTC+5:30, Adrien Grand wrote: > One difference is that the store is compressed while the structure of the > response that is built into memory is not (and potentially very wasteful). > > The doc-by-doc comparison is done by the post-filter: for every document > that matches the query, the filter is evaluated in order to know whether it > matches or not. On the other hand, when a filter is in the query (either > under a constant\_score or a filtered\_query), it can efficiently jump to the > next matches using the inverted index. > > If you want to export your index, a more efficient way would be to use the > `scan` search type: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html). > It basically opens a cursor that you can iterate on, as opposed as trying > to get everything at once. > > On Fri, Nov 21, 2014 at 4:43 PM, Ajay Divakaran \<[ajay.div...@gmail.com](mailto:ajay.div...@gmail.com) > \\> wrote: > > > Each document has around 350 text fields. > > > > But im still not able to relate the store size to the memory spike. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/c8014753-b0e2-481c-85dc-8a9eceae7d32%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c8014753-b0e2-481c-85dc-8a9eceae7d32%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/77ab05ed-39cd-4313-8d1a-9950995e8b71%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/77ab05ed-39cd-4313-8d1a-9950995e8b71%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [November 21, 2014, 6:08pm UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/9 "2014-11-21T18:08:35Z")
Seeing memory continuously increasing over a couple of requests is not necessarily a bad sign. If you give X gigabytes of memory to a JVM, it won't hesitate to use them if it can help decrease the frequency at which it has to run costly garbage collections. What is more important to watch is how memory usage behaves over a long period, eg. does the frequency at which GCs run keep on increasing (which could indicate that the server is encountering memory pressure or that there is something that leaks memory somewhere). On Fri, Nov 21, 2014 at 6:50 PM, Ajay Divakaran [ajay.divakaran86@gmail.com](mailto:ajay.divakaran86@gmail.com) wrote: > Thanks Adrien for your reply. > > I upgraded to ES 0.90.13. What I'm now noticing is that the memory seems > to continuously increase when running the query again and again. > I also noticed that after upgrading it started using OpenJDK 1.6\_0.33. So > I switched back to using Oracle JDK 1.7.71 however the issue seems to > persist. > > On Friday, November 21, 2014 10:57:09 PM UTC+5:30, Adrien Grand wrote: > > > One difference is that the store is compressed while the structure of the > > response that is built into memory is not (and potentially very wasteful). > > > > The doc-by-doc comparison is done by the post-filter: for every document > > that matches the query, the filter is evaluated in order to know whether it > > matches or not. On the other hand, when a filter is in the query (either > > under a constant\_score or a filtered\_query), it can efficiently jump to the > > next matches using the inverted index. > > > > If you want to export your index, a more efficient way would be to use > > the `scan` search type: [http://www.elasticsearch.org/](http://www.elasticsearch.org/) > > guide/en/elasticsearch/guide/current/scan-scroll.html. It basically > > opens a cursor that you can iterate on, as opposed as trying to get > > everything at once. > > > > On Fri, Nov 21, 2014 at 4:43 PM, Ajay Divakaran [ajay.div...@gmail.com](mailto:ajay.div...@gmail.com) > > wrote: > > > > > Each document has around 350 text fields. > > > > > > But im still not able to relate the store size to the memory spike. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/c8014753-b0e2-481c-85dc-8a9eceae7d32% > > > [40googlegroups.com](http://40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > Adrien Grand > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/77ab05ed-39cd-4313-8d1a-9950995e8b71%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/77ab05ed-39cd-4313-8d1a-9950995e8b71%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/77ab05ed-39cd-4313-8d1a-9950995e8b71%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/77ab05ed-39cd-4313-8d1a-9950995e8b71%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4zPn5ECjtcVCXjjmfNNbYyJ47HRsVdrvA%3DYHaoP-ZfbQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4zPn5ECjtcVCXjjmfNNbYyJ47HRsVdrvA%3DYHaoP-ZfbQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:48am UTC](https://discuss.elastic.co/t/term-filter-causes-memory-to-spike-drastically/20887/10 "2017-07-06T00:48:23Z")