### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 22, 2015, 5:48am UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/1 "2015-07-22T05:48:29Z")
Hello Elastic Experts,
I am reading article about out of memory during aggregation, for example, below articles, I do not quite understand why there is out of memory issue if we do filter + term aggregation? My feeling is after filter, the sub data set is small and term aggregation on that data set should not use too much memory. Looks like the out of memory is caused by Elastic Search is loading unnecessary data into memory during term aggregation -- if so what are the unnecessary data and how doc value helps here? Thanks.
>
> **[Disk-Based Field Data a.k.a. Doc Values
| Elastic](https://www.elastic.co/blog/disk-based-field-data-a-k-a-doc-values)**
>
> Elasticsearch is not just about full-text search, and many users are actually not using Elasticsearch for full-text search at all but for analytics though facets. This approach works well, but, as you...
thanks in advance,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 23, 2015, 7:23am UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/2 "2015-07-23T07:23:56Z")
Even with a filter you can still try to pull too much data into memory.
Are you having issues?
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 23, 2015, 7:55am UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/3 "2015-07-23T07:55:56Z")
Hi Mark,
Suppose I index all students in an index, I filter just by a single student ID (student ID are unique). Do you mean in this case, even if single student ID is simple filter, Elastic Search needs to load much more data into memory? If so, in this case, what are the much more data? 😄
regards,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 23, 2015, 8:02am UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/4 "2015-07-23T08:02:53Z")
No.
I mean that if you have millions or billions of records, even after filtering those down you can still have a lot of data, which can overwhelm the heap.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 23, 2015, 11:31pm UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/5 "2015-07-23T23:31:52Z")
Thanks Mark,
But how does DocValue helps here? I mean if after filter, we have millions or billions of record? My use case is, after filter, there are not too much records, but I am doing term aggregation on the results, and term vocabulary size may be around 100k. Your advice is highly appreciated.
regards,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 23, 2015, 11:37pm UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/6 "2015-07-23T23:37:15Z")
Doc values is fielddata but on disk, so it removes it from heap which gives you more space to do things like this.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 23, 2015, 11:51pm UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/7 "2015-07-23T23:51:35Z")
Thanks Mark,
If we want to do term aggregation, we need to put term (as a fielddata in an index) on disk as DocValue? Or we need to put other fielddata on disk as DocValue?
regards,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 23, 2015, 11:57pm UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/8 "2015-07-23T23:57:09Z")
Take a look at [https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html) and [https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html), it might explain it better.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 24, 2015, 3:55am UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/9 "2015-07-24T03:55:30Z")
Hi Mark,
Actually my question comes from the links. 😄
For term aggregation scenario, what are the differences between making term data field as DocValue or not?
BTW, you share the same documents twice? Or you want to share two different ones?
regards,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 24, 2015, 3:56am UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/10 "2015-07-24T03:56:20Z")
Doc values lives on disk.
Field data exists in heap.
The former is more efficient.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 24, 2015, 3:57am UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/11 "2015-07-24T03:57:28Z")
Thanks Mark,
But for term aggregation scenario, you mean make the term field data on disk?
regards,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 24, 2015, 4:11am UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/12 "2015-07-24T04:11:45Z")
Yes that is correct.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 24, 2015, 4:15am UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/13 "2015-07-24T04:15:37Z")
Thanks Mark,
In the document you referred, it is mentioned, "While in-memory fielddata has to be built on the fly at search time by uninverting the inverted index", what means uninverting the inverted index? Could you show an example of in the context of term aggregation, what does it mean?
regards,
Lin
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:59pm UTC](https://discuss.elastic.co/t/about-doc-values-feature-and-out-of-memory-during-term-aggregation/26038/14 "2017-07-05T23:59:21Z")