# Rebuilding cluster, few question about shards, index organization and retention **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2016, 8:40am UTC](https://discuss.elastic.co/t/rebuilding-cluster-few-question-about-shards-index-organization-and-retention/50066 "2016-05-15T08:40:07Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![matejzero](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@matejzero](https://discuss.elastic.co/u/matejzero)\ **Post date:** [May 15, 2016, 8:40am UTC](https://discuss.elastic.co/t/rebuilding-cluster-few-question-about-shards-index-organization-and-retention/50066/1 "2016-05-15T08:40:07Z")
Hello, I'm in the process of rebuilding our cluster and after some reading, I'm still not sure I get it right. Current status: 3 node cluster that is used for log storage from our servers. I process them with logstash and then save them to daily indexes, separating different log types(httpd,postfix,...) with type field. We create cca 10GB-18GB of logs per day and my indexes have 6 shards with 1 replica. What I want with a new cluster is better space eficiency, fastest indexing speed and some retention control. I did some reading and have sort of a plan in mind, but I'm not sure if I got everything right. Please, point out any mistakes. - Space efficency #1: Currently I have all logs in a single index, separated by type. As far as I read, if an index mapping has 100 fields, all those 100 fields are saved with every indexed document. So if I have a document with 10 fields, ES will still save all 100 fields, but most of them will be empty. Is that right? If yes, I was thinking of going with a single index per type. - Space efficency #2: If I will be separating logs to type per index, some daily indexes will be really small. I was thinking of moving to weekly or even monthly indexes. Currently, some indexes don't reach 100MB/day, while the biggest ones are around 5GB/day. If I go single index per month, would 150GB-200GB index be too big? What is a recommended index size? - Retention control: If I go with log type per index, it will be easier to maintain retention policy. When one type of logs are obsolete, I can just delete index and that's it. With the current setup, the only thing I can remember is Delete By Query which is slow or reindexing. Both require lots of work or resources. - Security: I want to limit access with reverse proxy. I read on your site that it's not a 100% safe option, but I think it will be good enough for us. Right now I'm using aliases for ACL, but it will be easier for us with type per index, since I can just limit/rewrite URL and give them access to certain type of logs. - Shards / replicas: I know this is the most frequently asked question:) How many shards and replicas. AFAIK ideally, you would want 1 shard per index, since every additional shard brings some overhead with it (ES has to fetch data from all nodes and then do mapreduce). But if I only have one shard, I can't scale horizontaly. I think I need some more info from you guys: I think that's it. I hope I'm not asking too basic questions. Thank you all, MAtej ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 16, 2016, 6:59am UTC](https://discuss.elastic.co/t/rebuilding-cluster-few-question-about-shards-index-organization-and-retention/50066/2 "2016-05-16T06:59:06Z")
[quote="matejzero, post:1, topic:50066"] What is a recommended shard size? I think I read somewhere that shard size should be less than amounth of HEAP SIZE (31GB in my case).[/quote] No, this has _never_ been the case. We suggest keeping shards to less than 50GB purely because moving more than that around for (re)allocation/replication is cumbersome. > Does more shards mean faster indexing speed? Yes, because you are writing small parts to many shards. > Does more shards mean faster searching speed? What about replicas, are they used in searching? Can more replicas make my searches faster? Depends, yes, yes. > If my index size is 5GB/month, is there any point in breaking it into more shards? I don't need the indexing performance, and I can take care of availability with replicas. No, a single shard is fine. > If my index size if 200GB/month and my first point is valid, recommended # of shards would be 6-7? I'd go with 4. > If I have multiple data.path on a single node, will one shard be saved in multiple data paths or will the whole shard be in a single path? The latter. > Lets say I have 2 paths/node and 3 nodes. If my index has 6 indices, ES will save 2 shards/node. But will it also save each shard on a node to a different data path if there is free space? It's not that smart. ---
**Author:** ![ddorian43](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddorian43/32/36093_2.png) [@ddorian43](https://discuss.elastic.co/u/ddorian43)\ **Post date:** [May 16, 2016, 10:49am UTC](https://discuss.elastic.co/t/rebuilding-cluster-few-question-about-shards-index-organization-and-retention/50066/3 "2016-05-16T10:49:09Z")
> [@matejzero](#): > > Security: I want to limit access with reverse proxy. I read on your site that it's not a 100% safe option, but I think it will be good enough for us. Right now I'm using aliases for ACL, but it will be easier for us with type per index, since I can just limit/rewrite URL and give them access to certain type of logs. If you say use nginx, and add ssl to it, and add htpasswd authentication, you should be fine. ---
**Author:** ![matejzero](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@matejzero](https://discuss.elastic.co/u/matejzero)\ **Post date:** [May 16, 2016, 1:30pm UTC](https://discuss.elastic.co/t/rebuilding-cluster-few-question-about-shards-index-organization-and-retention/50066/4 "2016-05-16T13:30:25Z")
Great! Thank you both for answers. One more thing I'm having troubles with. How many indexes / shards is too much (per cluster / per node). Is 600 indexes (each with 1 shard and 1 replica) too much (lets say I have 1 monthly index per log type, 10 different types and keep that data for 5 years). Older indexes are optimized at the end of the mont. Searches are mostly done inside the same log type, rarely span over different types. Thanks, Matej ---
**Author:** ![matejzero](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@matejzero](https://discuss.elastic.co/u/matejzero)\ **Post date:** [May 17, 2016, 2:52am UTC](https://discuss.elastic.co/t/rebuilding-cluster-few-question-about-shards-index-organization-and-retention/50066/5 "2016-05-17T02:52:50Z")
Another thing: From the resource usage point of view, is there any difference, if I have 1 index with 2 shards and 0 replicas or if I have 1 index with 1 shard and 1 replica? If I understand correctly, this are + and -: - 1 index with 2 shards and 0 replicas: - faster indexing - no HA - faster searching if data is in 2 different shards - 1 index with 1 shard and 2 replica: - slower indexing - HA - faster searching no matter where the data is ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 17, 2016, 5:55am UTC](https://discuss.elastic.co/t/rebuilding-cluster-few-question-about-shards-index-organization-and-retention/50066/6 "2016-05-17T05:55:10Z")
> [@matejzero](#): > > From the resource usage point of view, is there any difference, if I have 1 index with 2 shards and 0 replicas or if I have 1 index with 1 shard and 1 replica? No, a shard is a shard is a shard. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:51pm UTC](https://discuss.elastic.co/t/rebuilding-cluster-few-question-about-shards-index-organization-and-retention/50066/7 "2017-07-05T22:51:15Z")
Closed on July 5, 2017, 10:51pm UTC