# DocValues in ES 1.0.0.Beta1
**URL:** https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496
**Category:** Elasticsearch
**Created:** [November 20, 2013, 4:06pm UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496 "2013-11-20T16:06:37Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@ananth](https://discuss.elastic.co/u/ananth)
#### Post date: [November 20, 2013, 4:06pm UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/1 "2013-11-20T16:06:37Z")
Hi
Can any one help me how to use doc\_values in Beta1 , also how to verify the
same. I have the following settings in elasticsearch config directory.
_in elasticsearch.yml_
index.mapping.ignore\_malformed: false
index.store.type: mmapfs
index.codec.doc\_values\_format.my\_format.type: disk
_in templates section_
{
"app\_template" : {
"template" : "_2_",
"order" : 0,
"settings" : {
"index.number\_of\_shards" : 1,
"index.number\_of\_replicas" : 1
},
"mappings" : {
"2": {
"properties" : {
"time\_stamp" : {"type" : "long", "precision\_step": 0, "doc\_values\_format"
: "my\_format"},
"message" : {"type" : "string", "omit\_norms" : true, "index" : "analyzed"}
}}}}}
I have created a index with the above configuration and indexed few
documents . While searching i attached a debugger and found by default
doc\_value enabled for "\_version" field and "time\_stamp" field has null
value for docValueType . On looking in index directory \_0.cfe file contains
only one "doc value" file (both Lucene45\_0.dvd & \_Lucene45\_0.dvm , for
\_version field if i understand correctly) .
Then i removed index.codec.doc\_values\_format.my\_format.type: disk from
elasticsearch.yml and modified app\_template as below
"time\_stamp" : {"type" : "long", "precision\_step": 0, "doc\_values\_format" :
"disk"} but still the same result.
I indexed the same amount of data in ES 0.90.5(with out DocValue) and
issued a query . The field data stats api shows same amount of memory for
both ES 0.90.5 and ES 1 Beta (with DocValue).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@ananth](https://discuss.elastic.co/u/ananth)
#### Post date: [December 4, 2013, 11:53am UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/2 "2013-12-04T11:53:22Z")
DocValues working nicely . Thanks to this blog post.
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d06fe017-17d1-4473-b1e8-7e1deeda4d2c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d06fe017-17d1-4473-b1e8-7e1deeda4d2c%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [December 4, 2013, 8:03pm UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/3 "2013-12-04T20:03:21Z")
How would you define "working nicely"? Greater QPS? Reduced gc? What is
your metric? 🙂
I am curious about DocValues in storing norms as described by Simon almost
two years ago:
[Trifork Blog - Keep updated on the technical solutions Trifork is working on!](http://blog.trifork.com/2012/01/19/simon-says-single-byte-norms-are-dead/) The
loss of precision affects us somewhat since we take search relevancy
seriously.
Cheers,
Ivan
On Wed, Dec 4, 2013 at 3:53 AM, Anantha Govindarajan \<
[ananthagovindarajan@gmail.com](mailto:ananthagovindarajan@gmail.com)\> wrote:
> DocValues working nicely . Thanks to this blog post.
>
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/d06fe017-17d1-4473-b1e8-7e1deeda4d2c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d06fe017-17d1-4473-b1e8-7e1deeda4d2c%40googlegroups.com)
> .
>
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDj5F5JUUVvPeiDOCjS3V4axV0-POT7jaTeA3Z1R4nF5Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDj5F5JUUVvPeiDOCjS3V4axV0-POT7jaTeA3Z1R4nF5Q%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@ananth](https://discuss.elastic.co/u/ananth)
#### Post date: [December 6, 2013, 7:54am UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/4 "2013-12-06T07:54:06Z")
Hi Ivan , I meant doc\_values files .dvd & .dvm files are getting created
in index.
Initially i tried like following
"time\_stamp" : {"type" : "long", "precision\_step": 0, "doc\_values\_format" :
"my\_format"}.
Once come across this post i come to know that _doc\_values\_format_ conf
alone not enough to enable doc\_values , also field data format also needs
to be change like "fielddata" : { "format": "doc\_values"}. Now my template
looks following,
"time\_stamp" : {"type" : "long", "precision\_step": 0 , "fielddata" : {
"format": "doc\_values"}, "doc\_values\_format" : "disk"}.
Ivan , Thanks for asking this question , we yet to collect those kind of
metrics , i will share once i achieve those.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9d6f241b-69e4-43ec-8fa5-27b0686ebaf7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9d6f241b-69e4-43ec-8fa5-27b0686ebaf7%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jpountz](https://discuss.elastic.co/u/jpountz)
#### Post date: [December 7, 2013, 4:33pm UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/5 "2013-12-07T16:33:36Z")
Hi Anantha,
On Fri, Dec 6, 2013 at 8:54 AM, Anantha Govindarajan \<
[ananthagovindarajan@gmail.com](mailto:ananthagovindarajan@gmail.com)\> wrote:
> "time\_stamp" : {"type" : "long", "precision\_step": 0 , "fielddata" : {
> "format": "doc\_values"}, "doc\_values\_format" : "disk"}.
>
> Ivan , Thanks for asking this question , we yet to collect those kind of
> metrics , i will share once i achieve those.
>
> I see that you are using the "disk" doc-values format. This format tries
> to make the memory usage as low as possible, at some performance cost. My
> opinion is that the "default" doc values format is usually a better
> trade-off, since it only loads into memory the data-structures which are
> the most important performance-wise (which happen to be rather small).
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j78q8W\_9FZONnHCEbvkc8jn6U\_ojntRzUJjGqR2e2c9\_Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j78q8W_9FZONnHCEbvkc8jn6U_ojntRzUJjGqR2e2c9_Q%40mail.gmail.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@ananth](https://discuss.elastic.co/u/ananth)
#### Post date: [December 9, 2013, 5:06am UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/6 "2013-12-09T05:06:23Z")
Hi Adrien, Thanks for your suggestion. We are already facing memory issues
in ES data nodes. Some times GC collections are running more than 2
minutes. If i understand correctly due to GC, the node becomes unavailable
from the cluster , i.e. master mark it as down because node not responding
for all 3 retries with default 30 seconds interval (one & half minutes) ,
thats why we made it as disk doc\_value format. Any how let me check with
default doc\_value format also.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6888ff88-6181-4128-ab4e-02dc58fb1804%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6888ff88-6181-4128-ab4e-02dc58fb1804%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@otisg](https://discuss.elastic.co/u/otisg)
#### Post date: [December 12, 2013, 5:20am UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/7 "2013-12-12T05:20:56Z")
In our recent experiment (with Solr, not ES, but that doesn't matter here
because DocValues are Lucene goodness, not Solr or ES goodness) we saw a
much lower footprint with DocValues as we've used them.
Check slides 26-28 in
[http://blog.sematext.com/2013/11/09/presentation-solr-for-analytics/](http://blog.sematext.com/2013/11/09/presentation-solr-for-analytics/)
28 has a side by side heap comparison.
## Otis
Performance Monitoring \* Log Analytics \* Search Analytics
Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/)
On Wednesday, December 4, 2013 3:03:21 PM UTC-5, Ivan Brusic wrote:
> How would you define "working nicely"? Greater QPS? Reduced gc? What is
> your metric? 🙂
>
> I am curious about DocValues in storing norms as described by Simon almost
> two years ago:
> [http://blog.trifork.com/2012/01/19/simon-says-single-byte-norms-are-dead/](http://blog.trifork.com/2012/01/19/simon-says-single-byte-norms-are-dead/) The
> loss of precision affects us somewhat since we take search relevancy
> seriously.
>
> Cheers,
>
> Ivan
>
> On Wed, Dec 4, 2013 at 3:53 AM, Anantha Govindarajan \<
> [ananthago...@gmail.com](mailto:ananthago...@gmail.com) \\> wrote:
>
> > DocValues working nicely . Thanks to this blog post.
> >
> > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/)
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/d06fe017-17d1-4473-b1e8-7e1deeda4d2c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d06fe017-17d1-4473-b1e8-7e1deeda4d2c%40googlegroups.com)
> > .
> >
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/92af6eda-91fa-4835-af24-d174e0c28e57%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/92af6eda-91fa-4835-af24-d174e0c28e57%40googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@ananth](https://discuss.elastic.co/u/ananth)
#### Post date: [March 12, 2014, 1:00pm UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/8 "2014-03-12T13:00:44Z")
Hi Adrien ,
"My opinion is that the "default" doc values format is usually a better
trade-off, since it only loads into memory the data-structures which are
the most important performance-wise (which happen to be rather small). "
What are the important data-structures for doc\_values ? I am experiencing
improvements in response speed when using \*default \*instead of _disk_ doc\_value
format.
I have enabled default doc\_value format for long field(time\_stamp). What is
the difference between loading disk doc\_value and default doc\_value formats
?
Ananth
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7a9afc58-4d1a-4bcc-887c-1c00d6b39bda%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7a9afc58-4d1a-4bcc-887c-1c00d6b39bda%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jpountz](https://discuss.elastic.co/u/jpountz)
#### Post date: [March 12, 2014, 2:18pm UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/9 "2014-03-12T14:18:22Z")
Hi,
On Wed, Mar 12, 2014 at 2:00 PM, Anantha Govindarajan \<
[ananthagovindarajan@gmail.com](mailto:ananthagovindarajan@gmail.com)\> wrote:
> Hi Adrien ,
>
> "My opinion is that the "default" doc values format is usually a better
> trade-off, since it only loads into memory the data-structures which are
> the most important performance-wise (which happen to be rather small). "
>
> What are the important data-structures for doc\_values ? I am experiencing
> improvements in response speed when using \*default \*instead of _disk_ doc\_value
> format.
>
> I have enabled default doc\_value format for long field(time\_stamp). What
> is the difference between loading disk doc\_value and default doc\_value
> formats ?
Having the default format performing better than the disk format is
expected. The structures I'm talking about depend on the data type, but for
a long field (or numeric field in general), there are two structures: the
bytes of the numeric data (which are stored sequentially) and an index that
maps doc IDs to the offset where the numeric bytes are stored for that
particular document. For both "default" and "disk", the numeric bytes are
stored on disk, however "default" will store the index in memory while
"disk" will store it on disk. Please note that this index is efficiently
compressed, and should take very little memory, especially if all your
documents have the same number of values.
--
Adrien Grand
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4k%2Bf24CPftMzJKbROkJ-csMPCgNSyT6dsnaX-O1TR5VA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4k%2Bf24CPftMzJKbROkJ-csMPCgNSyT6dsnaX-O1TR5VA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:43am UTC](https://discuss.elastic.co/t/docvalues-in-es-1-0-0-beta1/14496/10 "2017-07-06T01:43:46Z")