# Doc values for field data **URL:** \ **Category:** Elasticsearch\ **Created:** [July 14, 2014, 5:26pm UTC](https://discuss.elastic.co/t/doc-values-for-field-data/18654 "2014-07-14T17:26:14Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![David\_Smith\_2](https://avatars.discourse-cdn.com/v4/letter/d/a9a28c/32.png) [@David\_Smith\_2](https://discuss.elastic.co/u/David_Smith_2)\ **Post date:** [July 14, 2014, 5:26pm UTC](https://discuss.elastic.co/t/doc-values-for-field-data/18654/1 "2014-07-14T17:26:14Z")
When you map fields to use doc values for field data, does that limit the functionality afforded to those fields to merely sorting and aggregations/faceting? The documentation mentions that filtering is not supported by numeric or string types when stored as doc values. Yikes, I thought that doc values is intended for working with field data when it's too large to load into memory. Is that not the case? I read both of the following pages but I'm not sure I quite understand where the usefulness of field data fields kick in. > **[Disk-Based Field Data a.k.a. Doc Values | Elastic](https://www.elastic.co/blog/disk-based-field-data-a-k-a-doc-values)** > > Elasticsearch is not just about full-text search, and many users are actually not using Elasticsearch for full-text search at all but for analytics though facets. This approach works well, but, as you... [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html) Can someone please clarify? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [July 15, 2014, 10:30am UTC](https://discuss.elastic.co/t/doc-values-for-field-data/18654/2 "2014-07-15T10:30:19Z")
Hi David, Doc values are a way to compute field data at indexing time, and to store it on disk. It can do everything that "uninverted" field data can do: aggregations, sorting, etc. However, it never kicks in automatically: it needs to be configured explicitely, and can only be set at index creation time, you cannot enable it afterwards. Regarding fielddata filtering, it is a way to trade accuracy for memory by only loading "important" terms into memory and doesn't work with doc values since it's not useful given that they are stored on disk anyway (and thus don't require much memory). Does it clarify? On Mon, Jul 14, 2014 at 7:26 PM, David K Smith [davidksmith2k@gmail.com](mailto:davidksmith2k@gmail.com) wrote: > When you map fields to use doc values for field data, does that limit the > functionality afforded to those fields to merely sorting and > aggregations/faceting? > > The documentation mentions that filtering is not supported by numeric or > string types when stored as doc values. Yikes, I thought that doc values is > intended for working with field data when it's too large to load into > memory. Is that not the case? > > I read both of the following pages but I'm not sure I quite understand > where the usefulness of field data fields kick in. > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/) > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html) > > Can someone please clarify? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7a68-vDManD3C\_TUXhB6jQxePhNcYx5VeFfku5AxuO2A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7a68-vDManD3C_TUXhB6jQxePhNcYx5VeFfku5AxuO2A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![David\_Smith\_2](https://avatars.discourse-cdn.com/v4/letter/d/a9a28c/32.png) [@David\_Smith\_2](https://discuss.elastic.co/u/David_Smith_2)\ **Post date:** [July 15, 2014, 1:25pm UTC](https://discuss.elastic.co/t/doc-values-for-field-data/18654/3 "2014-07-15T13:25:57Z")
Thanks, Adrien. That brings me closer. So when the documentations say doc values do not support filtering, it's talking about fielddata filtering for what's loaded into memory (anod not filtering as part of a query... say term filter). For further clarification - can a field that is not analyzed and only kept as doc values be used for querying/filtering (say a term filter on a numeric field or match query on a string field)? Or do all querying/filtering required the field to be in the uninverted index? What I'm trying to understand how we can optimize querying/filtering in a large index (5 billion documents / 1 TB)? It's very hard to run a simple term filter because a bitset filter will need to be calculated that includes every single document. Wouldn't that utilize a lot of memory? Is there a way to speed that up? On Tue, Jul 15, 2014 at 6:30 AM, Adrien Grand \< [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote: > Hi David, > > Doc values are a way to compute field data at indexing time, and to store > it on disk. It can do everything that "uninverted" field data can do: > aggregations, sorting, etc. However, it never kicks in automatically: it > needs to be configured explicitely, and can only be set at index creation > time, you cannot enable it afterwards. > > Regarding fielddata filtering, it is a way to trade accuracy for memory by > only loading "important" terms into memory and doesn't work with doc values > since it's not useful given that they are stored on disk anyway (and thus > don't require much memory). > > Does it clarify? > > On Mon, Jul 14, 2014 at 7:26 PM, David K Smith [davidksmith2k@gmail.com](mailto:davidksmith2k@gmail.com) > wrote: > > > When you map fields to use doc values for field data, does that limit the > > functionality afforded to those fields to merely sorting and > > aggregations/faceting? > > > > The documentation mentions that filtering is not supported by numeric or > > string types when stored as doc values. Yikes, I thought that doc values is > > intended for working with field data when it's too large to load into > > memory. Is that not the case? > > > > I read both of the following pages but I'm not sure I quite understand > > where the usefulness of field data fields kick in. > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/) > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html) > > > > Can someone please clarify? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/FC9E6ECA-B869-4B40-B2C8-F55CE6AB6790%40gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7a68-vDManD3C\_TUXhB6jQxePhNcYx5VeFfku5AxuO2A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7a68-vDManD3C_TUXhB6jQxePhNcYx5VeFfku5AxuO2A%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7a68-vDManD3C\_TUXhB6jQxePhNcYx5VeFfku5AxuO2A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7a68-vDManD3C_TUXhB6jQxePhNcYx5VeFfku5AxuO2A%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKoSUN87jc2nt8H07M%2BBxQuUcKCQPtsxdSL9S1Nf0cFe17EBFA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKoSUN87jc2nt8H07M%2BBxQuUcKCQPtsxdSL9S1Nf0cFe17EBFA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [July 16, 2014, 9:24am UTC](https://discuss.elastic.co/t/doc-values-for-field-data/18654/4 "2014-07-16T09:24:29Z")
On Tue, Jul 15, 2014 at 3:25 PM, David Smith [davidksmith2k@gmail.com](mailto:davidksmith2k@gmail.com) wrote: > Thanks, Adrien. That brings me closer. > > So when the documentations say doc values do not support filtering, it's > talking about fielddata filtering for what's loaded into memory (anod not > filtering as part of a query... say term filter). Exactly. > For further clarification - can a field that is not analyzed and only kept > as doc values be used for querying/filtering (say a term filter on a > numeric field or match query on a string field)? Or do all > querying/filtering required the field to be in the uninverted index? Doc values play no role when filtering (except for some filters that support a `fielddata` mode, such as the range filter[1]). So if your field has `index: no` you cannot use it in filters, and if it has `index: not_analyzed` then you can, no matter whether doc values are enabled or not. [1] > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. > What I'm trying to understand how we can optimize querying/filtering in a > large index (5 billion documents / 1 TB)? It's very hard to run a simple > term filter because a bitset filter will need to be calculated that > includes every single document. Wouldn't that utilize a lot of memory? Is > there a way to speed that up? If your filters are unlikely to be reused, then you should not cache them by setting \_cache to false. Caching filters only make filtering faster when the likelyhood of reusing filters is high. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6hE8CenTe9QfwWA5Rx45-mM%2BoOCSwPELOpsP\_tKTGthA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6hE8CenTe9QfwWA5Rx45-mM%2BoOCSwPELOpsP_tKTGthA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![David\_Smith\_2](https://avatars.discourse-cdn.com/v4/letter/d/a9a28c/32.png) [@David\_Smith\_2](https://discuss.elastic.co/u/David_Smith_2)\ **Post date:** [July 16, 2014, 2:07pm UTC](https://discuss.elastic.co/t/doc-values-for-field-data/18654/5 "2014-07-16T14:07:30Z")
Thank you, Adrien. That answers my questions. On Wednesday, July 16, 2014 5:24:36 AM UTC-4, Adrien Grand wrote: > On Tue, Jul 15, 2014 at 3:25 PM, David Smith \<[davidk...@gmail.com](mailto:davidk...@gmail.com) > \\> wrote: > > > Thanks, Adrien. That brings me closer. > > > > So when the documentations say doc values do not support filtering, it's > > talking about fielddata filtering for what's loaded into memory (anod not > > filtering as part of a query... say term filter). > > Exactly. > > > For further clarification - can a field that is not analyzed and only > > kept as doc values be used for querying/filtering (say a term filter on a > > numeric field or match query on a string field)? Or do all > > querying/filtering required the field to be in the uninverted index? > > Doc values play no role when filtering (except for some filters that > support a `fielddata` mode, such as the range filter[1]). So if your field > has `index: no` you cannot use it in filters, and if it has `index: not_analyzed` then you can, no matter whether doc values are enabled or not. > > [1] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-range-filter.html#_execution) > > > What I'm trying to understand how we can optimize querying/filtering in a > > large index (5 billion documents / 1 TB)? It's very hard to run a simple > > term filter because a bitset filter will need to be calculated that > > includes every single document. Wouldn't that utilize a lot of memory? Is > > there a way to speed that up? > > If your filters are unlikely to be reused, then you should not cache them > by setting \_cache to false. Caching filters only make filtering faster when > the likelyhood of reusing filters is high. > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c67ebf34-989b-4004-8b23-c9f7d00d9a13%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c67ebf34-989b-4004-8b23-c9f7d00d9a13%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:15am UTC](https://discuss.elastic.co/t/doc-values-for-field-data/18654/6 "2017-07-06T01:15:29Z")