# Doc Values vs Field Data Questions **URL:** \ **Category:** Elasticsearch\ **Created:** [May 21, 2015, 10:33pm UTC](https://discuss.elastic.co/t/doc-values-vs-field-data-questions/23666 "2015-05-21T22:33:00Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Matt\_Traynham](https://avatars.discourse-cdn.com/v4/letter/m/43a26b/32.png) [@Matt\_Traynham](https://discuss.elastic.co/u/Matt_Traynham)\ **Post date:** [May 21, 2015, 10:33pm UTC](https://discuss.elastic.co/t/doc-values-vs-field-data-questions/23666/1 "2015-05-21T22:33:00Z")
I'm doing some overall testing on my cluster, debating if I should switch to Doc Values. I have about 15 fields for each document, with 83 million documents spread across 60 indices. All the fields are dynamically mapped, and all of them can migrate to Doc Values. So, I have one copy of the data using FDC and a second copy using DV. Overall it's a 3x increase in consumed disk space, and a 98% decrease in FDC size when using DV. My question is, what is that last left over 2%? If everything is on disk, why is it reporting memory usage in the FDC? Some indices report 0 bytes, but others report anywhere between 34 KB - 700 KB? What am I missing here? Are things still loaded into the FDC anyway? Maybe I missed a field type in the dynamic templates? Also, what field types are recommended to move to Doc Values? High-cardinality non-analyzed string fields? High-cardinality in general? Everything? Overall performance seems similar, but this is just one of quite a few data-sets that would be interacted with at any given time and hopefully I have less memory issues (GC/eviction). ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2da6e886-5619-4313-a2a3-dcde99c986a2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2da6e886-5619-4313-a2a3-dcde99c986a2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Traynham](https://avatars.discourse-cdn.com/v4/letter/m/43a26b/32.png) [@Matt\_Traynham](https://discuss.elastic.co/u/Matt_Traynham)\ **Post date:** [May 21, 2015, 10:42pm UTC](https://discuss.elastic.co/t/doc-values-vs-field-data-questions/23666/2 "2015-05-21T22:42:58Z")
Just to correct myself, I misstated; a 1/3 increase in index size, not 3x. ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ab25d147-7775-4f48-bdeb-a798ffa9efeb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ab25d147-7775-4f48-bdeb-a798ffa9efeb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [May 21, 2015, 11:15pm UTC](https://discuss.elastic.co/t/doc-values-vs-field-data-questions/23666/3 "2015-05-21T23:15:52Z")
All fields that you search on and aggregate on should be moved to doc values in my opinion. By the way, elasticsearc 2.0 will make doc values on by default except on analyzed string fields. We still need some fielddata memory for something called the "global ordinal map". When you have string fields, we typically do computations using the ordinals of these strings instead of their actual values. However these ordinals only exist per segment, and sometimes we would like them to be consistent across an entire shard, so we build this global ordinal map that stores this mapping and keep it around. On Fri, May 22, 2015 at 12:42 AM, Matt Traynham [skitch920@gmail.com](mailto:skitch920@gmail.com) wrote: > Just to correct myself, I misstated; a 1/3 increase in index size, not 3x. > > ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) > > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ab25d147-7775-4f48-bdeb-a798ffa9efeb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ab25d147-7775-4f48-bdeb-a798ffa9efeb%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ab25d147-7775-4f48-bdeb-a798ffa9efeb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ab25d147-7775-4f48-bdeb-a798ffa9efeb%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAiz9kM9eKk%3Doh5AD9n6ZGFr7o4aTOYwC7ggC7jaK7%2BCRw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAiz9kM9eKk%3Doh5AD9n6ZGFr7o4aTOYwC7ggC7jaK7%2BCRw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Traynham](https://avatars.discourse-cdn.com/v4/letter/m/43a26b/32.png) [@Matt\_Traynham](https://discuss.elastic.co/u/Matt_Traynham)\ **Post date:** [May 22, 2015, 2:02pm UTC](https://discuss.elastic.co/t/doc-values-vs-field-data-questions/23666/4 "2015-05-22T14:02:16Z")
Thanks for the clarification Adrien. If that's the case, is there such a flag that can enable them by default for all fields (excluding non-analyzed strings; using ~1.4.3 here)? Also, do you guys have more performance metrics on using Doc Values vs FDC? I've seen the "10-25%" slower value thrown around, but I wanted to know what that was tested with (CPU, mem, spinning vs. SSD, etc...) and where gains may be had. ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8664452b-f3f4-45ac-bdfb-c41fbfc9cb7f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8664452b-f3f4-45ac-bdfb-c41fbfc9cb7f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Robert\_Muir\_3](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@Robert\_Muir\_3](https://discuss.elastic.co/u/Robert_Muir_3)\ **Post date:** [May 22, 2015, 2:36pm UTC](https://discuss.elastic.co/t/doc-values-vs-field-data-questions/23666/5 "2015-05-22T14:36:59Z")
On Fri, May 22, 2015 at 10:02 AM, Matt Traynham [skitch920@gmail.com](mailto:skitch920@gmail.com) wrote: > Thanks for the clarification Adrien. If that's the case, is there such a > flag that can enable them by default for all fields (excluding non-analyzed > strings; using ~1.4.3 here)? > > Also, do you guys have more performance metrics on using Doc Values vs > FDC? I've seen the "10-25%" slower value thrown around, but I wanted to > know what that was tested with (CPU, mem, spinning vs. SSD, etc...) and > where gains may be had. > > in my debugging the current differences are usually the cost of a > predictable branch (bounds check), coming from ByteBuffer.get(). For > fielddata it uses simple java arrays, and today the java compiler can do > optimizations to remove the checks more easily in that case. But IMO benchmarking here is usually not done correctly, it doesn't consider the impact of having such huge badly-compressed data in heap memory, e.g. impacts on GC and other problems people have. So i recommend doing a test with real data and real workloads 🙂 ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAD06sYSQ2U2y0D9ZOPMUDwywoXPYdAfq-6tDFsZOVKJ%3D0GatZA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAD06sYSQ2U2y0D9ZOPMUDwywoXPYdAfq-6tDFsZOVKJ%3D0GatZA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Matt\_Traynham](https://avatars.discourse-cdn.com/v4/letter/m/43a26b/32.png) [@Matt\_Traynham](https://discuss.elastic.co/u/Matt_Traynham)\ **Post date:** [May 25, 2015, 4:03pm UTC](https://discuss.elastic.co/t/doc-values-vs-field-data-questions/23666/6 "2015-05-25T16:03:11Z")
@rmuir Interesting, it sounds like my gains may be better than previously expected, given the server is constantly evicting from heap. If I'm able, I'll post some performance metrics back here when I'm done. ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f389ae24-9331-42b1-bd90-e0af617ffaee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f389ae24-9331-42b1-bd90-e0af617ffaee%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:12am UTC](https://discuss.elastic.co/t/doc-values-vs-field-data-questions/23666/7 "2017-07-06T00:12:04Z")