# With terms aggregations, should fields be marked store:true?

**URL:** <https://discuss.elastic.co/t/with-terms-aggregations-should-fields-be-marked-store-true/40096>\
**Category:** Elasticsearch\
**Created:** [January 26, 2016, 8:06am UTC](https://discuss.elastic.co/t/with-terms-aggregations-should-fields-be-marked-store-true/40096 "2016-01-26T08:06:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrew\_McFague](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@Andrew\_McFague](https://discuss.elastic.co/u/Andrew_McFague)\
**Post date:** [January 26, 2016, 8:06am UTC](https://discuss.elastic.co/t/with-terms-aggregations-should-fields-be-marked-store-true/40096/1 "2016-01-26T08:06:13Z")

</div>

I have billions of documents and am trying to do a Terms Aggregation on a string field. The field is not analyzed. However, when I try to perform the aggregation, it uses a significant amount of heap on my "aggregator" node, far more than should be required. I was able to perform similar aggregations on numeric fields with no issues.

Since it's not stored, it's my understanding that every time it looks at the field in question, it has to load the entirety of the full source back into memory so that it can read through the values for that given field. Is that true here as well? Or will it use the global ordinal once, and only look-up the value once-per-bucket? Likewise, if I mark the field as "store:true", should I see an increase in performance (and a decrease in heap usage)?

Thanks!

Andrew

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [January 27, 2016, 3:18pm UTC](https://discuss.elastic.co/t/with-terms-aggregations-should-fields-be-marked-store-true/40096/2 "2016-01-27T15:18:19Z")

</div>

Are you on a version of ES \< 2.0 ?

Unfortunately, this behavior is expected for string fields + field data. String fields use a lot of memory in pre-2.0 due to field data: strings themselves are largish (relative to numerics), and are often-multivalued, which requires a different data-structure which is a little less compressed in memory.

Post 2.0 we've switched to doc\_values by default. This is an on-disk, column store that requires _drastically_ less memory. Doc\_values run almost entirely "off-heap" (relying on OS cache), so you can drastically reduce your heap too.

You can use doc\_values in \<2.0 as well, but you have to enable it yourself.

Related, if you have multiply nested terms aggregations, you can get combinatorial explosion of "temporary" buckets. See this section of the Guide for more details and a solution: [Preventing Combinatorial Explosions | Elasticsearch: The Definitive Guide [2.x] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/current/_preventing_combinatorial_explosions.html)

Finally, `not_analyzed` strings tend to behave better than `analyzed` strings in field data, since they usually have lower cardinality as opposed to things like shingles, ngrams, etc. But sometimes, like with UUIDs or user names or product codes or IPv6, `not_analyzed` strings can have huge cardinality. Higher cardinality increases memory usage in field data.

> [@Andrew\_McFague](#):
>
> Since it's not stored, it's my understanding that every time it looks at the field in question, it has to load the entirety of the full source back into memory so that it can read through the values for that given field. Is that true here as well?

When you set a field to `not_analyzed`, it is still indexed into the inverted index...it just doesn't go through the analysis phase. So when you execute an aggregation, the data is loaded from the inverted index, not the original source, so you don't have to worry about re-parsing the original JSON, touching disk `n` times for each doc, etc.

You're correct about the global ordinals though, once the terms are populated into memory from the inverted index they will generate ordinals for the segments and the global ordinal map, and use that. As you update/add documents though, parts or all of the ordinal map will need to be refreshed.

> [@Andrew\_McFague](#):
>
> Likewise, if I mark the field as "store:true", should I see an increase in performance (and a decrease in heap usage)?

Nope! But don't worry, it's a very common misconception 🙂

Stored fields simply take the original `_source` value for that field and stick it in a new field. It doesn't affect the inverted index or aggregations at all, it simply saves the pre-analysis version of the field separately for retrieval.

It's main use-case is when you have a 5mb PDF and want to display the `title` field to the user: you'd prefer not to parse the 5mb JSON just to get the title. So saving it separately can save some disk IO and parsing time.

---

<div class="post-metadata">

**Author:** ![Andrew\_McFague](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@Andrew\_McFague](https://discuss.elastic.co/u/Andrew_McFague)\
**Post date:** [January 27, 2016, 7:58pm UTC](https://discuss.elastic.co/t/with-terms-aggregations-should-fields-be-marked-store-true/40096/3 "2016-01-27T19:58:23Z")

</div>

Thank you so much for the explanation, Zachary!

I actually already switched to doc\_values some time ago, because the buckets were so large they were overloading the heap on my data nodes and causing the entire server to come to a grinding halt! But based on your explanation of how "store" works, it doesn't sound like there would be much of an impact from my original question.

I've also specifically made all my string fields `not_analyzed` (by design--I genuinely don't want them analyzed). Fortunately, there aren't a lot of new ordinals being added, especially not in bulk.

In this case, I'm using a string because performing an aggregation on multiple numeric values is expensive, because it requires new ordinals to be created on the fly (i.e., via a script), or to an intermediate string (copy\_to). In this case, I have "long:long", when I could have multiple keys instead. Is there any way to take advantage of using multiple doubles as a "single" ordinal for buckets aggregation?

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [January 28, 2016, 8:04pm UTC](https://discuss.elastic.co/t/with-terms-aggregations-should-fields-be-marked-store-true/40096/4 "2016-01-28T20:04:17Z")

</div>

> [@Andrew\_McFague](#):
>
> In this case, I have "long:long", when I could have multiple keys instead. Is there any way to take advantage of using multiple doubles as a "single" ordinal for buckets aggregation?

Hmm, not really. One thing you could try is, instead of building the combined tuple, just embed a few terms aggs together. E.g. instead of making the tuple `foo:bar`, you could do:

```auto
{
  "terms": {
    "field" : "foo",
    "collect_mode" : "breadth_first"
  },
  "aggs": {
    "terms": {
      "field" : "bar"
    }
  }
}

```

The advantage here is that you can leverage `breadth_first` to prune the search tree aggressively, and avoid building tuples on the fly. It'll require a bit more processing in your app (since the data is now split over two aggs, instead of a single agg), but it could be faster / less memory.

Alternatively, `copy_to` might be a legit strategy if this is a common operation. Adding that data straight into the index will help performance tremendously (for relatively little impact on storage requirements)

Or if you don't need to know the actual tuple itself (and can tolearate a potential collision), you could hash the value and store that, which will be faster because it can be stored as a long instead of a string

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:20pm UTC](https://discuss.elastic.co/t/with-terms-aggregations-should-fields-be-marked-store-true/40096/5 "2017-07-05T23:20:39Z")

</div>