# Storage/Performance optimization
**URL:** https://discuss.elastic.co/t/storage-performance-optimization/152393
**Category:** Elasticsearch
**Created:** [October 14, 2018, 3:31pm UTC](https://discuss.elastic.co/t/storage-performance-optimization/152393 "2018-10-14T15:31:22Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Neha\_Sharma1](https://discuss.elastic.co/u/Neha_Sharma1)
#### Post date: [October 14, 2018, 3:31pm UTC](https://discuss.elastic.co/t/storage-performance-optimization/152393/1 "2018-10-14T15:31:23Z")
We receive approx. 1TB of data everyday(including replica) for 100+ indexes. We currently have 16TB storage spread which stores data for 15 days. Out ES cluster has 2 dedicates MI nodes and 6 Di nodes and 1 MDI node. (M = master D = data I = Ingest)
We would like to optimize our cluster for better storage (of 30 days) and index/search performance.
1. Should we have dedicated master nodes(with no ingestion)?
2. Should we add client nodes for query purposes?
3. Can we use best\_compression for storage? how would it affect my search queries?
4. We create a snapshot of the current index state every day. In case we plan to restore index for any day which is around 150 Gb, it takes approx 2 hours to restore. Is there a faster way to restore indexes from the snapshot?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 14, 2018, 4:40pm UTC](https://discuss.elastic.co/t/storage-performance-optimization/152393/2 "2018-10-14T16:40:33Z")
How many indices/shards do you have in your cluster? Based on your description it sounds like you may be having a lot of small indices and shards, [which can be very inefficient](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster).
> [@Neha\_Sharma1](#):
>
> Should we have dedicated master nodes(with no ingestion)?
Yes. Allowing the dedicated master nodes to just manage the cluster is recommended as best practice as it improves cluster stability. You should also make sure you have 3 of them.
> [@Neha\_Sharma1](#):
>
> Should we add client nodes for query purposes?
There is no clear recommendation on this. It depends a lot on your use case.
> [@Neha\_Sharma1](#):
>
> Can we use best\_compression for storage? how would it affect my search queries?
It generally does not affect searching, but will require a bit extra work at indexing time.
> [@Neha\_Sharma1](#):
>
> We create a snapshot of the current index state every day. In case we plan to restore index for any day which is around 150 Gb, it takes approx 2 hours to restore. Is there a faster way to restore indexes from the snapshot?
The defaults for restoring snapshots are set to not overwhelm the cluster, but can be tweaked. I believe the restore process uses the shard recovery mechanism, so you can increase the [indices.recovery.max\_bytes\_per\_sec](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/cluster-update-settings.html#cluster-update-settings) setting to make it more aggressive. You should also be able to alter the [shard allocation settings](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/shards-allocation.html#_shard_allocation_settings).
---
### Author:  [@Neha\_Sharma1](https://discuss.elastic.co/u/Neha_Sharma1)
#### Post date: [October 14, 2018, 5:16pm UTC](https://discuss.elastic.co/t/storage-performance-optimization/152393/3 "2018-10-14T17:16:24Z")
Thanks [Christian\_Dahlqvist]
We have over 100 micro-services, so each micro-service has its own index. We have default shard number set to 6. Our index size differs from few MBs to 500-660 GB.
Also currently some of our nodes in the cluster have primary shards only. Is there a setting which needs to be tuned so that all primary shards in the cluster are equally distributed on nodes and so are the replicas after that.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 14, 2018, 7:03pm UTC](https://discuss.elastic.co/t/storage-performance-optimization/152393/4 "2018-10-14T19:03:09Z")
That sounds quite inefficient. I would recommend changing the shard count according to the index sizes to better align with the recommendations in the blog post I linked to. If you primarily are indexing new data and not updating documents, primaries and replicas basically do the same work, so an uneven distribution of primary shards may not matter much.
---
### Author:  [@Neha\_Sharma1](https://discuss.elastic.co/u/Neha_Sharma1)
#### Post date: [October 18, 2018, 7:23am UTC](https://discuss.elastic.co/t/storage-performance-optimization/152393/5 "2018-10-18T07:23:12Z")
But we have observed that:
As the indexes are time-series, all of them get created at 12 AM UTC. The problem is that all the primary shards of most of the indexes are allocated to one node(say N1) and few primary shards and replicas are assigned to other nodes.
Cluster configuration set to:
cluster.routing.rebalance.enable: all
cluster.routing.allocation.allow\_rebalance: indices\_all\_active
thread\_pool.bulk.queue\_size: 1000
During bulk indexing requests, all of the requests go to that node (N1) and CPU utilization increases for this node. A lot of requests are also rejected as the queue size exceeds on that node. Whereas other nodes stay chilled out.
Doubts:
1. Is the above issue, is it because all the primary shards are on one node only?
2. If yes, Can I rebalance my primary shards by setting "cluster.routing.rebalance.enable" to "primaries". Would this configuration first rebalance my primary shards and then balance the replicas? Are there any repercussions.
3. Is there any other cause of the issue and is there a way to mitigate it?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 18, 2018, 7:33am UTC](https://discuss.elastic.co/t/storage-performance-optimization/152393/6 "2018-10-18T07:33:02Z")
> [@Neha\_Sharma1](#):
>
> Is the above issue, is it because all the primary shards are on one node only?
Not necessarily.
> [@Neha\_Sharma1](#):
>
> If yes, Can I rebalance my primary shards by setting "cluster.routing.rebalance.enable" to "primaries". Would this configuration first rebalance my primary shards and then balance the replicas? Are there any repercussions.
As far as I know there is no way to efficiently control this as Elasticsearch need to manage this in response to index and cluster events.
> [@Neha\_Sharma1](#):
>
> Is there any other cause of the issue and is there a way to mitigate it?
As outlined in [this blog post](https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster), writing to lots of shards can be inefficient and quickly fill up bulk indexing queues. Ways to improve this would be to dramatically reduce the number of shards being written to and/or try to ensure each bulk request target a smaller set of indices/shards, e.g. through the use of [multiple discreet pipelines](https://www.elastic.co/guide/en/logstash/6.4/multiple-pipelines.html) and [pipeline-to-pipeline communication](https://www.elastic.co/guide/en/logstash/6.4/pipeline-to-pipeline.html).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 15, 2018, 7:33am UTC](https://discuss.elastic.co/t/storage-performance-optimization/152393/7 "2018-11-15T07:33:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.