# Fluctuating Index Sizes

**URL:** <https://discuss.elastic.co/t/fluctuating-index-sizes/69262>\
**Category:** Elasticsearch\
**Created:** [December 16, 2016, 8:32am UTC](https://discuss.elastic.co/t/fluctuating-index-sizes/69262 "2016-12-16T08:32:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\
**Post date:** [December 16, 2016, 8:32am UTC](https://discuss.elastic.co/t/fluctuating-index-sizes/69262/1 "2016-12-16T08:32:44Z")

</div>

Hi,  
We're seeing significant fluctuations in index sizes in our log aggregation cluster for old indices that are no longer being updated. For example, the index for Dec 12th doubled in size in the days after its last update before reducing in size by around 15%.

Can someone help explain what is/may be going on here, how we can track the processes at work and what we can do to deal with it?

Regards,  
David

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 19, 2016, 1:17am UTC](https://discuss.elastic.co/t/fluctuating-index-sizes/69262/2 "2016-12-19T01:17:09Z")

</div>

It could be due to merging segments, with sparse docvalues it can increase the size of the segments.

---

<div class="post-metadata">

**Author:** ![dawiro](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dawiro](https://discuss.elastic.co/u/dawiro)\
**Post date:** [December 20, 2016, 7:44am UTC](https://discuss.elastic.co/t/fluctuating-index-sizes/69262/3 "2016-12-20T07:44:55Z")

</div>

Hi,  
The thing is we optimise all of our indices after 1 day and the index I'm referring to here was growing after 3 days. Ended up being 2.2TB in size based on around 500 million docs before settling back to 1.7TB. When the last write came in the index was around 950GB!

BTW, is there a way of measuring the sparsity of our field data? Would I need to run lucene commands to do that? We're running on 2.1.2...

Regards,  
David

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [December 20, 2016, 10:19am UTC](https://discuss.elastic.co/t/fluctuating-index-sizes/69262/4 "2016-12-20T10:19:50Z")

</div>

You can run an exists query [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html) on a given field and compare the result with the number of docs that you have in your index.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2017, 10:19am UTC](https://discuss.elastic.co/t/fluctuating-index-sizes/69262/5 "2017-01-17T10:19:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.