# Advantages of DocValue
**URL:** https://discuss.elastic.co/t/advantages-of-docvalue/26271
**Category:** Elasticsearch
**Created:** [July 24, 2015, 9:53pm UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271 "2015-07-24T21:53:33Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 24, 2015, 9:53pm UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/1 "2015-07-24T21:53:33Z")
Hello Elastic Search experts,
The advantages of performance of DocValue is able to leverage OS disk cache, which is different (maybe more) memory from JVM. Wondering for non-DocValue field, it is only able to use memory of JVM itself (if so, why it is not able to leverage OS disk cache for performance boosting, which could avoid us explicitly specify physical storage when defining logical index structure)?
> **[Disk-Based Field Data a.k.a. Doc Values
| Elastic](https://www.elastic.co/blog/disk-based-field-data-a-k-a-doc-values)**
>
> Elasticsearch is not just about full-text search, and many users are actually not using Elasticsearch for full-text search at all but for analytics though facets. This approach works well, but, as you...
thanks in advance,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 25, 2015, 12:35am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/2 "2015-07-25T00:35:51Z")
Non doc values, ie fielddata, is held in heap so it'll be pretty fast.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 25, 2015, 1:02am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/3 "2015-07-25T01:02:43Z")
Thanks Mark,
For non doc values, they can only use Java heap? Cannot leverage OS level buffer?
regards,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 25, 2015, 2:10am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/4 "2015-07-25T02:10:57Z")
They aren't on disk, so no.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 25, 2015, 2:34am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/5 "2015-07-25T02:34:18Z")
Thanks Mark to clarify,
Then for Doc Values, how do they leverage memory outside of JVM? Doc Value are still JVM (Java) data structures (if so, how do Doc Value fields leverage memory outside JVM)?
regards,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 25, 2015, 5:04am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/6 "2015-07-25T05:04:10Z")
They use the OS cache, which uses any free system memory to hold files. This is why we recommend setting heap to 50% of total system RAM.
Doc values are not JVM structures though.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 25, 2015, 5:51am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/7 "2015-07-25T05:51:39Z")
Thanks Mark,
If Doc Values are not JVM data structures, what are they using and where are they? JNI is still using JVM memory I think? Please feel free to correct me if I am wrong.
Have a good weekend. 😄
regards,
Lin
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 25, 2015, 8:46am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/8 "2015-07-25T08:46:32Z")
Doc values are mapped into the process's address space instead of (like field data) being read into memory allocated from the JVM heap.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 26, 2015, 5:01am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/9 "2015-07-26T05:01:29Z")
Thanks Mark,
Are there any more details how "Doc values are mapped into the process's address"? I tried Doc Value and it works great, just curious to learn a bit more. 😄
Have a great weekend.
regards,
Lin
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 26, 2015, 8:28am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/10 "2015-07-26T08:28:03Z")
Not sure exactly what you're looking for, but Wikipedia's article on [memory-mapped files](https://en.wikipedia.org/wiki/Memory-mapped_file) seems to explain things pretty well. Any book on operating systems (e.g. Tanenbaum) also covers this.
---
### Author:  [@upayavira](https://discuss.elastic.co/u/upayavira)
#### Post date: [July 26, 2015, 3:53pm UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/11 "2015-07-26T15:53:10Z")
Lucene's main data structure is an inverted index. That is, terms point
to documents.
For things like sorting and faceting, this doesn't work, because you
need to be able to point from a document to a term (if you want to sort
by the price field, you need to identify the value of the price field
for a specific document).
We solve this with an uninverted index, such as Lucene's FieldCache. The
field cache is built in the background by Lucene by reading through the
inverted index on each commit, and "uninverting" it. This takes an on-
disk data structure, which (on certain OSes) can be accessed via a memory-
mapped file system, and creates an in-heap data structure. This can be
really fast, but suffers from the need to build the data structure
entirely on each commit, which can take some time.
DocValues provide a solution. They are an uninverted column based store,
that is build at index time as an on-disk data structure.
The point of the memory-mapped filesystem is quite simple. Lucene
developers noticed that when they were loading indexes into memory,
they were loading them from disk into the OS disk cache, and then from
there into the Java heap. As a result, there were two copies of the
data in memory, which was overkill. The solution was to switch to using
memory-mapped files in which Java can access the files in the OS disk
cache as if they were simply in memory, thus halving memory
requirements for Lucene.
Upayavira
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 26, 2015, 9:53pm UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/12 "2015-07-26T21:53:25Z")
Thanks Upayavira for the details,
1. For your comments below, want to confirm my understanding is correct. Java still access through Java File I/O interface as if accessing a normal disk file, but underlying the files are already in memory (which is using OS cache memory other than JVM memory) using memory mapped file?
> [@upayavira](#):
>
> The solution was to switch to using
> memory-mapped files in which Java can access the files in the OS disk
> cache as if they were simply in memory
1. For your comments, "We solve this with an uninverted index", confused what means an uninverted index? In your sample, I think it could be implemented simply by the revert index (which is price field =\> document ID), suppose we want to find document whose price is more than 10 USD, we just scan the invert index price field and keep those only above 10 USD? Please feel free to correct me if I am wrong. 😄
BTW, when you say "faceting", do you mean filtering by a field?
regards,
Lin
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 26, 2015, 10:00pm UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/13 "2015-07-26T22:00:52Z")
Thanks Magnus,
In the scenario of filter student information (suppose each student as a document) by student ID filed. I think Elastic Search will build revert index from student ID to the real student information document. For memory mapped file in this use case, the memory mapped file is used on revert index (student ID =\> student information document)? Or used on the student information document itself?
regards,
Lin
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 26, 2015, 10:02pm UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/14 "2015-07-26T22:02:25Z")
He's Magnus, I'm Mark.
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [July 26, 2015, 10:05pm UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/15 "2015-07-26T22:05:27Z")
Corrected, 😉
Both if you are super experts. 😄
regards,
Lin
---
### Author:  [@linlma](https://discuss.elastic.co/u/linlma)
#### Post date: [August 6, 2015, 3:44am UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/17 "2015-08-06T03:44:01Z")
@upayavira, it will be great if you could comment on my questions. 😄
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:57pm UTC](https://discuss.elastic.co/t/advantages-of-docvalue/26271/18 "2017-07-05T23:57:13Z")