# Data directory is still growing after initial index build **URL:** \ **Category:** Elasticsearch\ **Created:** [January 18, 2018, 2:24pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075 "2018-01-18T14:24:02Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![4b3l](https://avatars.discourse-cdn.com/v4/letter/4/9de0a6/32.png) [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Post date:** [January 18, 2018, 2:24pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/1 "2018-01-18T14:24:02Z")
Hi, I have a 3 Node ES Cluster. I have built an index in my new cluster. This took a while to complete. I've noticed that since the index being built, the data directory is still growing and I am not sure why or where to troubleshoot this, our Application is idle so there aren't any requests going through. Anyone have any ideas or point me in the right direction? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 18, 2018, 3:13pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/2 "2018-01-18T15:13:12Z")
Might be segment merge is happening behind the scene? Monitoring (available in X-Pack basic - free) could help to see that may be. ---
**Author:** ![4b3l](https://avatars.discourse-cdn.com/v4/letter/4/9de0a6/32.png) [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Post date:** [January 22, 2018, 11:21am UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/3 "2018-01-22T11:21:41Z")
Thanks for your reply. Not sure which specific monitoring to use, I used \_cat/segments?v and \_segments, but it doesn't tell me if there are any active segment merging going on I don't see anything in the X-pack API documentation. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 22, 2018, 11:44am UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/4 "2018-01-22T11:44:09Z")
X-Pack monitoring collects information about the segment numbers. And lot of other metrics. You should give it a try. ---
**Author:** ![4b3l](https://avatars.discourse-cdn.com/v4/letter/4/9de0a6/32.png) [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Post date:** [January 22, 2018, 2:28pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/5 "2018-01-22T14:28:38Z")
Thanks. From reading the documentation Segment merging is something that happens constantly behind the scenes but is triggered when there is new data coming in. Our system has been idle for over a week, e.g. no new data going in. Seems strange it takes over a week for something like this to happen or is it just how Lucene behaves? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 22, 2018, 2:56pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/6 "2018-01-22T14:56:35Z")
> Our system has been idle for over a week, e.g. no new data going in. Then it looks strange indeed. Could you run: ```auto GET _cat/indices?v ``` ---
**Author:** ![4b3l](https://avatars.discourse-cdn.com/v4/letter/4/9de0a6/32.png) [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Post date:** [January 22, 2018, 3:16pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/7 "2018-01-22T15:16:37Z")
The list is big to copy and paste? Is there anything to look for? health is green, and status open for all indexes. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 22, 2018, 4:24pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/8 "2018-01-22T16:24:18Z")
You can share it on [gist.github.com](http://gist.github.com) and just add the link here. ---
**Author:** ![4b3l](https://avatars.discourse-cdn.com/v4/letter/4/9de0a6/32.png) [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Post date:** [January 22, 2018, 4:28pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/9 "2018-01-22T16:28:14Z")
> [@dadoonet](#): > > \_cat/indices?v thanks: Here is the url: > ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 22, 2018, 4:33pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/10 "2018-01-22T16:33:15Z")
Can you explain what are all those indices? Can you run the same command again and compare which one is still growing? As you are using xpack, for sure the number of documents in indices like `.monitoring-es-*` is still increasing. But it should not be that much. ---
**Author:** ![4b3l](https://avatars.discourse-cdn.com/v4/letter/4/9de0a6/32.png) [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Post date:** [January 22, 2018, 4:54pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/11 "2018-01-22T16:54:11Z")
Just ran it again it seems the same, I will do this again after 1 hour to confirm. The indices are our data we have created our own naming convention. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 22, 2018, 5:11pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/12 "2018-01-22T17:11:58Z")
Ok but that's 600 indices on 3 nodes. That is becoming to be a lot. And probably a waste of resources as you have at most some hundred of mb used per shard. Also notice that the big indices you have are the monitoring ones (some gb). ---
**Author:** ![4b3l](https://avatars.discourse-cdn.com/v4/letter/4/9de0a6/32.png) [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Post date:** [January 23, 2018, 10:02am UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/13 "2018-01-23T10:02:17Z")
I just did a compare and the monitoring-es indices seem to grow to a max 5.7GB/5.8GBbefore a new one is created on each day, reading some background it keeps up to 7 days for a basic license. What are the purpose of these indices. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 23, 2018, 10:16am UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/14 "2018-01-23T10:16:59Z")
> What are the purpose of these indices. They are used by X-Pack Monitoring. You can change `xpack.monitoring.collection.interval` (defaults to every `10s`). See [Monitoring Settings in Elasticsearch | Elasticsearch Reference [6.1] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/monitoring-settings.html#monitoring-collection-settings) Also you can change `xpack.monitoring.history.duration` to `1d` so you should keep only one day of monitoring data online. Anyway, that's why your disk space is increasing. ---
**Author:** ![4b3l](https://avatars.discourse-cdn.com/v4/letter/4/9de0a6/32.png) [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Post date:** [January 23, 2018, 11:55am UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/15 "2018-01-23T11:55:05Z")
Thank you for your help and patience 😀 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 23, 2018, 12:17pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/16 "2018-01-23T12:17:41Z")
You're very welcomed. Just a note. Let me highlight what I said previously as you might have troubles in the future: > [@dadoonet](#): > > Ok but that's 600 indices on 3 nodes. That is becoming to be a lot. > > And probably a waste of resources as you have at most some hundred of mb used per shard. ---
**Author:** ![4b3l](https://avatars.discourse-cdn.com/v4/letter/4/9de0a6/32.png) [@4b3l](https://discuss.elastic.co/u/4b3l)\ **Post date:** [January 23, 2018, 2:42pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/17 "2018-01-23T14:42:08Z")
Is there a guideline of how many indices we should have or is there an article around this subject? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 23, 2018, 3:03pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/18 "2018-01-23T15:03:47Z")
May I suggest you look at the following resources about sizing: [https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing) > **[How many shards should I have in my Elasticsearch cluster? | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet... > **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)** > > Surveiller une application complexe n’est pas une tĂąche aisĂ©e, mais avec les bons outils, ce n’est pas si sorcier. NĂ©anmoins, des pĂ©riodes fortes telles que les opĂ©rations de type « Black Friday » (Vendredi noir) ou pĂ©riode de NoĂ«l peuvent pousser... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 20, 2018, 3:03pm UTC](https://discuss.elastic.co/t/data-directory-is-still-growing-after-initial-index-build/116075/19 "2018-02-20T15:03:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.