# Indexing performance with doc values (particularly with larger number of fields) **URL:** \ **Category:** Elasticsearch\ **Created:** [March 24, 2014, 2:01am UTC](https://discuss.elastic.co/t/indexing-performance-with-doc-values-particularly-with-larger-number-of-fields/16554 "2014-03-24T02:01:08Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Alex\_At\_Ikanow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_at_ikanow/32/676_2.png) [@Alex\_At\_Ikanow](https://discuss.elastic.co/u/Alex_At_Ikanow)\ **Post date:** [March 24, 2014, 2:01am UTC](https://discuss.elastic.co/t/indexing-performance-with-doc-values-particularly-with-larger-number-of-fields/16554/1 "2014-03-24T02:01:08Z")
This might be more of a Lucene question, but a quick google didn't throw up anything. Has anyone done/seen any benchmarking on indexing performance (overhead) due to using doc values? I often index quite large JSON objects, with many fields (eg 50), I'm trying to get a feel for whether I can just let all of them be doc values on the off chance I'll want to aggregate over them, or whether I need to pick beforehand which fields will support aggregation. (A related question: presumably allowing a mix of doc values fields and "legacy" fields is a bad idea, because if you use doc values fields you want a low max heap so that the file cache has lots of memory available, whereas if you use the field cache you need a large heap - is that about right, or am i missing something?) Thanks for any insight! Alex Ikanow -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0361eda4-ab39-4536-b91a-ccb710921edd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0361eda4-ab39-4536-b91a-ccb710921edd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Robert\_Muir\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_muir_2/32/931_2.png) [@Robert\_Muir\_2](https://discuss.elastic.co/u/Robert_Muir_2)\ **Post date:** [March 24, 2014, 2:26am UTC](https://discuss.elastic.co/t/indexing-performance-with-doc-values-particularly-with-larger-number-of-fields/16554/2 "2014-03-24T02:26:34Z")
Would be a nice benchmark to run (and if you find hotspots/slow things to go improve in lucene...)! The data structures for docvalues are less complex than the data structures for the inverted index. I've enabled docvalues for many fields as you suggest in the past, and in my tests the time for e.g. segment merging was still dominated by the inverted index (terms dict, postings lists, etc), as I had all the fields indexed for search, too. But nothing is free: some of this stuff is data-dependent so you have to test. About the heap, you are right, its probably best to adjust your heap accordingly if you are using dovalues. On Sun, Mar 23, 2014 at 10:01 PM, Alex at Ikanow [apiggott@ikanow.com](mailto:apiggott@ikanow.com) wrote: > This might be more of a Lucene question, but a quick google didn't throw up > anything. > > Has anyone done/seen any benchmarking on indexing performance (overhead) due > to using doc values? > > I often index quite large JSON objects, with many fields (eg 50), I'm trying > to get a feel for whether I can just let all of them be doc values on the > off chance I'll want to aggregate over them, or whether I need to pick > beforehand which fields will support aggregation. > > (A related question: presumably allowing a mix of doc values fields and > "legacy" fields is a bad idea, because if you use doc values fields you want > a low max heap so that the file cache has lots of memory available, whereas > if you use the field cache you need a large heap - is that about right, or > am i missing something?) > > Thanks for any insight! > > Alex > Ikanow > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/0361eda4-ab39-4536-b91a-ccb710921edd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0361eda4-ab39-4536-b91a-ccb710921edd%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAMUKNZVGxGcM\_QrFHEXsaa%3DQcH\_Er\_h1s4LgBQDE0kU7c%2Bi2JQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAMUKNZVGxGcM_QrFHEXsaa%3DQcH_Er_h1s4LgBQDE0kU7c%2Bi2JQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:41am UTC](https://discuss.elastic.co/t/indexing-performance-with-doc-values-particularly-with-larger-number-of-fields/16554/3 "2017-07-06T01:41:16Z")