# Indices size and # of shard
**URL:** https://discuss.elastic.co/t/indices-size-and-of-shard/163278
**Category:** Elasticsearch
**Created:** [January 7, 2019, 10:31pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278 "2019-01-07T22:31:07Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [January 7, 2019, 10:31pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/1 "2019-01-07T22:31:08Z")
After lot of testing I am going to move to production.
Is it ok to have default 5 shard and large indices size 50gig with 150Million documents?
cluster I have setup is pretty big 8 node with three master and all 8 node acting as data node as well.
I might have 10 more of this type of large indices on this cluster with 50 more small indices.
I want to get better idea before I dump all this in to production. I know I don't have any disk space issue or memory issue as all my systems has good amount of memory and disk space.
I just want to make sure I won't get big performance issue with this type of setup.
All three master server is acting as logstash and kibana server as well.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 8, 2019, 7:41am UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/2 "2019-01-08T07:41:38Z")
How many indices/shards are you planning to create every day? How long do you intend to keep data in the cluster? What is the specification of your Elasticsearch nodes?
I would recommend having a look at the following resources:
> **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
> **[Optimizing Storage Efficiency in Elasticsearch](https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch)**
>
> This video covers the different types of nodes we use in Hot/Warm/Cold architectures and discuss their characteristics and the factors that determine how much data each node type can hold and how you go about optimizing for this.
> [@elasticforme](#):
>
> All three master server is acting as logstash and kibana server as well.
This in my opinion sounds like a bad idea unless you have very powerful nodes. Even in that case I would put the additional processes on nodes that are not master eligible. Elasticsearch assumes all nodes are equal by default and here you are taking resources away from the Elasticsearch nodes you want to be the most stable.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [January 8, 2019, 2:20pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/3 "2019-01-08T14:20:11Z")
all systems has 98 gig ram, three of them are on 10gig network, rest of them are on gig network.
some of the storage are 10k disk, some are 7.2k. most of them are dual socket four core cpus
No new indices will be created that often. just updated existing indices, each has five shard. going to keep only one year of data and delete once a month older data from indices using delete\_by\_query.
Three system in USA, three in UK and three in Singapore.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 8, 2019, 2:29pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/4 "2019-01-08T14:29:22Z")
OK, so you will have 3 separate clusters?
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [January 8, 2019, 4:24pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/5 "2019-01-08T16:24:04Z")
one cluster, but spread out across reagion.
I just checked my biggest indice (19.2gig) largest shard for that indice is 4.4gig. I do not expect that to be more then double even if I keep two year of record. This size is just one year record 2018.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 8, 2019, 4:30pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/6 "2019-01-08T16:30:44Z")
Elasticsearch requires low latencies between nodes, so deploying a cluster across data centres that far apart is not recommended nor supported. I would therefore recommend setting up a separate cluster per region.
The following blog posts may be of interest:
> **[Clustering Across Multiple Data Centers](https://www.elastic.co/blog/clustering_across_multiple_data_centers)**
>
> This post answers the question: Can I run an Elasticsearch cluster across multiple data centers? Includes detailed reasons and alternatives.
> **[Tribe Nodes & Cross-Cluster Search: The Future of Federated Search in...](https://www.elastic.co/blog/tribe-nodes-and-cross-cluster-search-the-future-of-federated-search-in-elasticsearch)**
>
> Elasticsearch has a powerful \_search API that allows it to search against all indices on the local cluster. We recently released Elasticsearch 5.3.0 including a new functionality called Cross-Cl...
> **[Scaling Elasticsearch Across Data Centers With Kafka](https://www.elastic.co/blog/scaling_elasticsearch_across_data_centers_with_kafka)**
>
> The article proposes architectures to support scaling across data centers using Elasticsearch and Kafka.
> **[Elasticsearch 6.5.0 Released
| Elastic](https://www.elastic.co/blog/elasticsearch-6-5-0-released)**
>
> Today we are pleased to announce the release of Elasticsearch 6.5.0, based on Lucene 7.5.0. This is the latest stable release, and is already available for deployment via our Elasticsearch Service on ...
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [January 8, 2019, 7:21pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/7 "2019-01-08T19:21:05Z")
Thank you Christian
looks like design point of view I have to rethink and redo cluster
from indice/shard point of view I think I am ok.
---
### Author:  [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)
#### Post date: [January 8, 2019, 8:24pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/8 "2019-01-08T20:24:08Z")
The recommended shard size ~8-10GB so that relocation of shards of initialization of shards if they go unassigned is fast. So for a 50GB sized index, # of shards can be around ~6.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 8, 2019, 8:47pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/9 "2019-01-08T20:47:24Z")
If you are going to keep data in the cluster for a long time, try to aim for an average shard size of between 30GB and 50GB. In your initial description it sounded like you were going to create a good number of smaller indices, which I would recommend against.
---
### Author:  [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [January 17, 2019, 9:35pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/10 "2019-01-17T21:35:23Z")
you can close this. Large Cluster is broken in to separate clusters, shard size is not more then 4.4 gig. and max size for Indice is 41gb. Even if I go two year record it will be 80gig and shard size will be 8 gig.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 14, 2019, 9:35pm UTC](https://discuss.elastic.co/t/indices-size-and-of-shard/163278/11 "2019-02-14T21:35:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.