(StreamInput.java:246)
```
Only master node blows up while data nodes are okay.
What's happening in master's memory? ðŸ˜
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 21, 2016, 11:46pm UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/6 "2016-02-21T23:46:07Z")
Shards are not free and carries a certain amount of overhead with respect to memory and file handles. With that many indices, the cluster state is also likely to be quite large and use up a fair amount of memory.
Having 78000 (if I count correctly) shards is way, way too many for a cluster of that size and specification, and will use up a lot of memory. I recommend you rethink your indexing/sharding strategy in order to dramatically reduce the number of shards in the cluster.
---
### Author:  [@dynamicscope](https://discuss.elastic.co/u/dynamicscope)
#### Post date: [February 22, 2016, 12:17am UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/7 "2016-02-22T00:17:57Z")
Thank you very much for your answer.
I have been creating indices as daily log style, e.g. `my-index-001-20160221` for handy management. And 001 is ranging from 001 ~ 100.
I guess this is a bad indexing strategy for memory usage.
Q. What if I disable norm for the fields? I am only filtering and aggregating for many of the fields. Would this help solving memory issue?
Q. What could be the side effect if I change my index pattern to be `my-index-001-YYYYMM`? This will reduce the number of shards by `1/30`-ish but I would need to update/change daily docs when a day ends.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 22, 2016, 12:24am UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/8 "2016-02-22T00:24:36Z")
Why are you creating 100 indices per day? Your shards at the moment appear to be extremely small, containing an average of less than 3000 documents if I calculated correctly, which is very inefficient. Without knowing the answer to my initial question, I would recommend creating a single monthly index with 3 shards. Why would this force you to to update/change daily docs when a day ends?
---
### Author:  [@dynamicscope](https://discuss.elastic.co/u/dynamicscope)
#### Post date: [February 22, 2016, 12:35am UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/9 "2016-02-22T00:35:11Z")
I have separated indices by users. So more users will create more index per day. And I chose such strategy in order to handle when I no longer need a single user's data.
Also, If I put all users data in a single index, there is going to be more search/aggregation requests to a single index. And I was worrying about this would give me slow responses.
And I daily removes and backup to S3 one field that contains a large value (in order to save some storage space).
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 22, 2016, 12:50am UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/10 "2016-02-22T00:50:25Z")
> [@dynamicscope](#):
>
> I have separated indices by users. So more users will create more index per day. And I chose such strategy in order to handle when I no longer need a single user's data.
This will generally scale badly. You can use a single index for all users and use [routing by the customer ID](https://www.elastic.co/blog/customizing-your-document-routing) for both indexing and querying to ensure just a single shard per index need to be accessed. This can help with query throughput and latency.
Elasticsearch can handle a lot of data in an index/shard and still return quick responses, so I would recommend benchmarking it. Having larger shards will also allow Elasticsearch to more efficiently compress you data on disk.
---
### Author:  [@dynamicscope](https://discuss.elastic.co/u/dynamicscope)
#### Post date: [February 22, 2016, 12:59am UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/11 "2016-02-22T00:59:45Z")
Thank you very much.
If I had to choose either.
1. Monthly index for every user. eg. `my-index-001-YYYYMM`
2. Daily index for all users. eg. `my-index-YYYYMMDD`
3. Monthly index for all users. eg. `my-index-YYYYMM`
You would recommend 2 rather than 1. And 3 at the best? Or 2 should be fine to solve my issue.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 22, 2016, 1:05am UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/12 "2016-02-22T01:05:35Z")
Reducing the number of indices by a factor of 100 would be a big win. Whether to go for monthly or daily indices depend on how long your retention period is. If it is shorter than a couple of months I would consider daily indices. For longer retention periods monthly indices will allow your cluster to handle larger amounts of data.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 22, 2016, 1:11am UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/13 "2016-02-22T01:11:24Z")
That is WAY WAY WAY WAY too many shards.
You really need to reduce them as Christian mentioned.
---
### Author:  [@dynamicscope](https://discuss.elastic.co/u/dynamicscope)
#### Post date: [February 22, 2016, 5:36am UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/14 "2016-02-22T05:36:21Z")
I had to stop the cluster since it was unresponsive.
And I've deleted half of my index directories, then restarted the cluster.
Odd thing I notice from restarting the cluster. The number of pending tasks increase very fast.
And throws the following after an hour even before cluster health go green.
> [2016-02-22 02:43:18,508][WARN][monitor.jvm] [Hulk] [gc][old][4055][48] duration [30.4s], collections [1]/[31.4s], total [30.4s]/[37.4s], memory [9.8gb]-\>[9.6gb]/[9.9gb], all\_pools {[young] [207mb]-\>[7.6mb]/[266.2mb]}{[survivor] [33.2mb]-\>[0b]/[33.2mb]}{[old] [9.6gb]-\>[9.6gb]/[9.6gb]}
Q. Could a lot of pending tasks can cause GC to fail and make the cluster unresponsive?
(This time I don't see OOM error.)
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [February 22, 2016, 5:08pm UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/15 "2016-02-22T17:08:01Z")
> [@dynamicscope](#):
>
> What could I do to make my cluster back alive?
See our previous comments.
---
### Author:  [@dynamicscope](https://discuss.elastic.co/u/dynamicscope)
#### Post date: [February 22, 2016, 5:15pm UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/16 "2016-02-22T17:15:07Z")
Force-deleting half of index directories didn't help making the cluster back alive.
I tried doubling the master node's memory, but it still dies.
My initial plan was to make the cluster back alive and re-indexing everything.
(with `my-index-YYYYMM` pattern)
Since the cluster doesn't easily get back on, I plan to create new cluster and find a way to restore data from the dead cluster. (Not sure this is possible at the moment.)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:14pm UTC](https://discuss.elastic.co/t/what-happends-if-i-change-master-true-data-true-node-to-master-only-node/42337/17 "2017-07-05T23:14:26Z")