# Elasticsearch clustering and storage issues **URL:** \ **Category:** Elasticsearch\ **Created:** [April 2, 2020, 1:03pm UTC](https://discuss.elastic.co/t/elasticsearch-clustering-and-storage-issues/226216 "2020-04-02T13:03:51Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![snambu5878](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/snambu5878/32/45746_2.png) [@snambu5878](https://discuss.elastic.co/u/snambu5878)\ **Post date:** [April 2, 2020, 1:03pm UTC](https://discuss.elastic.co/t/elasticsearch-clustering-and-storage-issues/226216/1 "2020-04-02T13:03:51Z")
Hi Everyone, We are doing a POC to analyse and visualize 64 million records (175 colomns) of data on a 3 node cluster. Below are the settings: Each node capacity: CPU: 10 cores; RAM: 64Gig; Storage: 150GB I am using logstash to ingest data from Netezza to Elasticsearch. After the logstash is started, we have updated replica of the index to 0 as this is a test. We are getting lot of low disk errors even though the total cluster storage is 450 GB (total storage of cluster). individual node disk space is 150GB. I have checked the index size for instance 12.5 million records took around 50GB of space on single node. I have the following questions: 1. Can you please let us know how the storage is split across cluster? For Example, if the disk space on one node is low, will elasticsearch stores the data in other nodes? 2. Any better recommendations around ingesting the data into the cluster. 3. What the next steps in case the node goes read only mode. Will appreciate your quick response on this. Thanks, Srikanth ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [April 2, 2020, 1:43pm UTC](https://discuss.elastic.co/t/elasticsearch-clustering-and-storage-issues/226216/2 "2020-04-02T13:43:51Z")
> [@snambu5878](#): > > We are getting lot of low disk errors What are they? ---
**Author:** ![Ian\_Boje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ian_boje/32/52033_2.png) [@Ian\_Boje](https://discuss.elastic.co/u/Ian_Boje)\ **Post date:** [April 2, 2020, 2:09pm UTC](https://discuss.elastic.co/t/elasticsearch-clustering-and-storage-issues/226216/3 "2020-04-02T14:09:25Z")
> [@snambu5878](#): > > Can you please let us know how the storage is split across cluster? For Example, if the disk space on one node is low, will elasticsearch stores the data in other nodes? Each index can have multiple shards. Those shards can be balanced across all of your nodes automatically. If you're running out of space, I'd suspect you have a single shard taking up too much space on a single node. Do you have multiple indexes and/or several shards per index configured? Some documentation on shard balancing: > **[Disk-based shard allocation | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html)** My summary of this: watermark.low - I can take more shards watermark.high - I'm not taking any more shards watermark.flood\_stage - I'm going to move my shards to other nodes > [@snambu5878](#): > > What the next steps in case the node goes read only mode. I struggle with this at times too. I've found that [Cerebro](https://github.com/lmenezes/cerebro) is a handy tool to troubleshoot issues. Under More-\>Cluster settings, there are some read-only switches that you might be enabled when your disk fills. ---
**Author:** ![snambu5878](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/snambu5878/32/45746_2.png) [@snambu5878](https://discuss.elastic.co/u/snambu5878)\ **Post date:** [April 2, 2020, 6:15pm UTC](https://discuss.elastic.co/t/elasticsearch-clustering-and-storage-issues/226216/4 "2020-04-02T18:15:28Z")
Hi Ian thanks for you quick reply. ```auto As per elastic, Default for number_of_shards is 5, Default for number_of_replicas is 1 (ie one replica for each primary shard). That means i have 5 shards and not multiple indexes. I am creating only one index (400 gig) because I have a requirement to visualize full data. We cannot create visualization referring to multiple indexes right? For ex: Dec 2019 data in one index and Nov 2019 data in another index then create a visualization for Nov and Dev 2019 month with different indexes. This is not possible right? Also, my Kibana is down with apm plugin not available error suddenly not sure why but node2 and node3 are up and running whereas node 1 is down due to disk full issues. Appreciate your help. ``` Thanks, Srikanth ---
**Author:** ![snambu5878](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/snambu5878/32/45746_2.png) [@snambu5878](https://discuss.elastic.co/u/snambu5878)\ **Post date:** [April 2, 2020, 6:17pm UTC](https://discuss.elastic.co/t/elasticsearch-clustering-and-storage-issues/226216/5 "2020-04-02T18:17:23Z")
Hi Dadoonet, Thanks for you reply. These are because the disk space is full on node1. I have been loading 64 million records to elasticsearch and my understanding is if the disk space on node 1 is 85%, elasticsearch will relocate the data to multiple other nodes. Not sure what settings i need to do to acheive this. Appreciate your help. Thanks, Srikanth ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 30, 2020, 6:17pm UTC](https://discuss.elastic.co/t/elasticsearch-clustering-and-storage-issues/226216/6 "2020-04-30T18:17:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.