Hello all, Setting up an elasticsearch cluster for the first time here. I need some advice on creating my index. This index will store information about emails. The plan is to retain about 10 years worth of message data. I performed some analysis on the logs and it looks like the largest month we …

Awesome, thanks @warkolm and @elasticforme for the input, this helps a lot.

welcome back joshua, 250gig per month, lets divide by 4 (weekly index) = 62gig, per index (index_name-year-01 to 52 ) and each index with two shard, each is 30gig. (well within limit) you will have 52 index * 2 = 104 shard, 104 replica - per year 10 year you will have 1040 shard and 1040 repli…

A single index for time based data does not make sense, use time based indices :slight_smile:

Thank you for the advice, super helpful!

Yeah, I think I'm starting to see that. So using the example that elasticforme gave, what would be the benefit/con of using 4 shards (or 8, or 100) per weekly index vs 2? Can you help me understand the tradeoff? I think that's really the meat of the question.

more shard will use more memory, cluster need to manage all that shard more is not good. and bigger then 30gig shard is not good either. so many smaller shard is not good either. it won't give you any speed.

[image] elasticforme: ger then 30gig shard is not good either. It's not? Why?

all the document in elk says it. :slight_smile:

I think you will find that is in reference to heap size, not shard size.

TLDR - it depends. On your ingest rate, your query rate, the sorts of queries you do on the size of the doc. I would start with a ~40 gig shard size and go from there. It may make sense to increase shard size for archived data for eg. I would strongly recommend the use of ILM as well.

Need advice on shards for my index

Elastic Stack Elasticsearch

jcannell (Joshua Cannell) September 1, 2020, 9:29pm 13

You can't directly specify shard size, right? This would be based on the number of shards and how the amount of data in the index is divided, correct? So a 100GB index with 4 shards = 25gb/shard?

Topic		Replies	Views
Recommended # of Shards for 1.5 TB = 2.752.250.197 Docs Elasticsearch	6	1368	March 29, 2018
Index with few shard or index with many shards? Elasticsearch	10	665	January 21, 2019
How many shards should I create in terms of an index across 20 data node? Elasticsearch	6	3817	July 21, 2017
Is Daily-Index better than Monthly-Index Elasticsearch	6	2044	May 26, 2020
Shard Recommendation for Elasticsearch Elasticsearch	4	353	July 6, 2017

Need advice on shards for my index

Related topics