# Disk quota on Elasticsearch for indices from different sources? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 10, 2017, 8:47pm UTC](https://discuss.elastic.co/t/disk-quota-on-elasticsearch-for-indices-from-different-sources/81870 "2017-04-10T20:47:40Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Mike\_Z](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_z/32/122216_2.png) [@Mike\_Z](https://discuss.elastic.co/u/Mike_Z)\ **Post date:** [April 10, 2017, 8:47pm UTC](https://discuss.elastic.co/t/disk-quota-on-elasticsearch-for-indices-from-different-sources/81870/1 "2017-04-10T20:47:40Z")
We have a self-hosted Elasticsearch cluster, accepting log feeds from multiple sources. In a recent incident, issues on one single source caused entire cluster failure and data loss to all the other sources. Due to a Logstash configuration error, one feeding source suddenly went crazy by sending a huge amount of data within a short period. We have a very big disk buffer, but it was still eaten up. All the nodes got a disk full, and the cluster’s functionality came into a halt. It took some time to fix the problem, and incoming data is lost during the gap. Feels like, a single point of issue should not fail the entire cluster, and we want to look into possibly restricting total disk usage of indices from each source. On our cluster, each source creates one index per day, with naming pattern source\_name-YYYY-MM-DD, and we can get total disk usage per source by command “du -chs /indices/source\_name-\* | grep total”. Tentatively thinking, maybe we can have a watchdog script to check it, and close or delete the indices that exceed the quota. I am wondering if there is any existing tool for it? I am also wondering how Elasticsearch Cloud handles disk usage issue, assuming there are also multiple clients on their shared host. Is it deploying one virtual machine for each client or something similar? Feedbacks and pointers will be highly appreciated. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 11, 2017, 8:24am UTC](https://discuss.elastic.co/t/disk-quota-on-elasticsearch-for-indices-from-different-sources/81870/2 "2017-04-11T08:24:45Z")
> [@Mike\_Z](#): > > I am wondering if there is any existing tool for it? I haven't seen something for this specific use case. Generally retention is cluster/node wide. ---
**Author:** ![Mike\_Z](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_z/32/122216_2.png) [@Mike\_Z](https://discuss.elastic.co/u/Mike_Z)\ **Post date:** [April 11, 2017, 3:15pm UTC](https://discuss.elastic.co/t/disk-quota-on-elasticsearch-for-indices-from-different-sources/81870/3 "2017-04-11T15:15:42Z")
Thank you for your input, Mark. I am also wondering how the cloud service is handling the disk space issue, assuming there are also multiple clients on the shared host. Is it deploying one virtual machine for each client or something similar? More details would be greatly helpful to me. ---
**Author:** ![Mike\_Z](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_z/32/122216_2.png) [@Mike\_Z](https://discuss.elastic.co/u/Mike_Z)\ **Post date:** [April 11, 2017, 5:48pm UTC](https://discuss.elastic.co/t/disk-quota-on-elasticsearch-for-indices-from-different-sources/81870/4 "2017-04-11T17:48:34Z")
Found a related post on GitHub, including the link below just for your reference. > ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 11, 2017, 10:47pm UTC](https://discuss.elastic.co/t/disk-quota-on-elasticsearch-for-indices-from-different-sources/81870/5 "2017-04-11T22:47:42Z")
Cloud does it per node, which is a container. So really it's handled as it would be for any other node. ---
**Author:** ![Mike\_Z](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_z/32/122216_2.png) [@Mike\_Z](https://discuss.elastic.co/u/Mike_Z)\ **Post date:** [April 12, 2017, 9:48pm UTC](https://discuss.elastic.co/t/disk-quota-on-elasticsearch-for-indices-from-different-sources/81870/6 "2017-04-12T21:48:43Z")
Thank you for your help, Mark. With multiple feeding sources, is it more popular to use separated nodes facing each source respectively? We have security requirements. If change to this direction, is there a way to build an internal cloud service, by Docker or what? Or, is there a product for this usage? > [@warkolm](#): > > Cloud does it per node, which is a container. So really it's handled as it would be for any other node ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 13, 2017, 3:42am UTC](https://discuss.elastic.co/t/disk-quota-on-elasticsearch-for-indices-from-different-sources/81870/7 "2017-04-13T03:42:00Z")
Elastic Cloud Enterprise does this use case very well. However you could DIY with docker too. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 11, 2017, 3:52am UTC](https://discuss.elastic.co/t/disk-quota-on-elasticsearch-for-indices-from-different-sources/81870/8 "2017-05-11T03:52:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.