# Is sparse data on not indexed field affect on ES performance?

**URL:** <https://discuss.elastic.co/t/is-sparse-data-on-not-indexed-field-affect-on-es-performance/112771>\
**Category:** Elasticsearch\
**Created:** [December 21, 2017, 8:49am UTC](https://discuss.elastic.co/t/is-sparse-data-on-not-indexed-field-affect-on-es-performance/112771 "2017-12-21T08:49:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![majid\_zarrin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/majid_zarrin/32/29103_2.png) [@majid\_zarrin](https://discuss.elastic.co/u/majid_zarrin)\
**Post date:** [December 21, 2017, 8:49am UTC](https://discuss.elastic.co/t/is-sparse-data-on-not-indexed-field-affect-on-es-performance/112771/1 "2017-12-21T08:49:34Z")

</div>

I have sparse data based on a filed named type.  
assume

> if (type = A ) fields x and y appears  
> if(type=B) fields w and z appears.  
> if(type=C) no extra fields appears ( about 70% of data ~ 2 billion data )

I know that sparsity affects on Elastic search performance.  
I planned to overcome this problem by setting sparse fields index to false and create another index for indexing them.  
is my strategy true?  
thank you.

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [December 21, 2017, 2:40pm UTC](https://discuss.elastic.co/t/is-sparse-data-on-not-indexed-field-affect-on-es-performance/112771/2 "2017-12-21T14:40:30Z")

</div>

Spasity affects mostly:

- norms, which are on by default on indexed text fields
- doc values, which are on by default on keyword, date, ip and numeric fields

> [@majid\_zarrin](#):
>
> I planned to overcome this problem by setting sparse fields index to false and create another index for indexing them.
> 
> is my strategy true?

I don't think it will help, then your other index will be sparse?  
If you really need to support sparse data, I'd suggest that you use Elasticsearch 6.x which has much better support for sparsity.

---

<div class="post-metadata">

**Author:** ![animageofmine](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@animageofmine](https://discuss.elastic.co/u/animageofmine)\
**Post date:** [January 9, 2018, 6:43pm UTC](https://discuss.elastic.co/t/is-sparse-data-on-not-indexed-field-affect-on-es-performance/112771/3 "2018-01-09T18:43:57Z")

</div>

@jpountz What is the cost of sparsity? Is it disk space only assuming norms and doc\_values are enabled? Or does this consume heap space as well?

Meaning, if there are too many sparse fields across documents in an index, would it consume additional disk space compared to the same index with dense data?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 6, 2018, 6:44pm UTC](https://discuss.elastic.co/t/is-sparse-data-on-not-indexed-field-affect-on-es-performance/112771/4 "2018-02-06T18:44:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.