# In ES \>=6.0, is sparsity for doc\_values & norms still bad? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 23, 2018, 8:19am UTC](https://discuss.elastic.co/t/in-es-6-0-is-sparsity-for-doc-values-norms-still-bad/132946 "2018-05-23T08:19:55Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![janzankowski](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@janzankowski](https://discuss.elastic.co/u/janzankowski)\ **Post date:** [May 23, 2018, 8:19am UTC](https://discuss.elastic.co/t/in-es-6-0-is-sparsity-for-doc-values-norms-still-bad/132946/1 "2018-05-23T08:19:55Z")
Hello, ES 6.0 and up uses Lucene 7, where [doc values and norms have switched from random access API to iterator API](http://blog.mikemccandless.com/2017/03/apache-lucene-70-is-coming-soon.html). What this means, as I understand it, is that if a document does not have a given field that other docs in the index have, we'll no longer have to pay in disk space for that field. Given the above, is the advice in ES General Recommendations, under heading [Avoid Sparsity](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/general-recommendations.html#sparsity), still relevant? It's still there in documentation for ES \>=6.0, and says things like: ``` In practice, this means that if an index has M documents, norms will require M bytes of storage per field, even for fields that only appear in a small fraction of the documents of the index. Although slightly more complex with doc values due to the fact that doc values have multiple ways that they can be encoded depending on the type of field and on the actual data that the field stores, the problem is very similar. ``` Thank you, Jan ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 23, 2018, 8:33am UTC](https://discuss.elastic.co/t/in-es-6-0-is-sparsity-for-doc-values-norms-still-bad/132946/2 "2018-05-23T08:33:44Z")
[This blog post](https://www.elastic.co/blog/minimize-index-storage-size-elasticsearch-6-0) provides some additional detail and guidelines around the change. ---
**Author:** ![janzankowski](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@janzankowski](https://discuss.elastic.co/u/janzankowski)\ **Post date:** [May 23, 2018, 1:59pm UTC](https://discuss.elastic.co/t/in-es-6-0-is-sparsity-for-doc-values-norms-still-bad/132946/3 "2018-05-23T13:59:42Z")
Thanks for that. So, you would confirm that the specific bit of explanation why sparsity is bad - the one I quote above from the 6.0 docs - is now obsolete, right? If so, where can I file a request to have it removed? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 23, 2018, 2:02pm UTC](https://discuss.elastic.co/t/in-es-6-0-is-sparsity-for-doc-values-norms-still-bad/132946/4 "2018-05-23T14:02:56Z")
I am not sure which part that are still valid, but you can open an issue against the [docs repo on GitHub](https://github.com/elastic/docs) for a clarification or change. ---
**Author:** ![janzankowski](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@janzankowski](https://discuss.elastic.co/u/janzankowski)\ **Post date:** [May 24, 2018, 8:37am UTC](https://discuss.elastic.co/t/in-es-6-0-is-sparsity-for-doc-values-norms-still-bad/132946/5 "2018-05-24T08:37:37Z")
Submitted [https://github.com/elastic/elasticsearch/issues/30833](https://github.com/elastic/elasticsearch/issues/30833) (Against elastic/elasticsearch, not elastic/docs - the latter says "If you find an error in the documentation, you should open an issue or pull request on the repository which contains the docs. For instance, the elasticsearch docs can be found in the main elasticsearch repository.") ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 21, 2018, 8:37am UTC](https://discuss.elastic.co/t/in-es-6-0-is-sparsity-for-doc-values-norms-still-bad/132946/6 "2018-06-21T08:37:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.