# Best Cluster Concept, Timeseries Data and more **URL:** \ **Category:** Elasticsearch\ **Tags:** ilm-index-lifecycle-management\ **Created:** [July 16, 2020, 2:23pm UTC](https://discuss.elastic.co/t/best-cluster-concept-timeseries-data-and-more/241482 "2020-07-16T14:23:10Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\ **Post date:** [July 16, 2020, 2:23pm UTC](https://discuss.elastic.co/t/best-cluster-concept-timeseries-data-and-more/241482/1 "2020-07-16T14:23:10Z")
Hi, I am currently developing some cluster concept and once again asking for your opinions and insight. We will need different clusters for different customers. I have some questions regarding my concepts and I hope you can help with this. **Concept 1** , Simple Cluster. Should be able to handle **200MB/day**. Maybe more in the future. What should I do when I add **new nodes**? The elastic documentation states that after 3 Nodes I should start with giving some nodes **specific rolles**. The problem that I have with this is that I want tho have a reliable cluster so when I start to add specific rolles I will need **at least 3 new nodes** with these roles so that they are failproof. Right? So when I want nodes that are only Master I will need 3 new Nodes which only do the master job. That doesn't sound to good. ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/3/937048979e8d1a40ad129f1ed966b4d89dbe7a47.png) **Concept 2** , Master/Data: This concept should be used when we have customers with more data. **20GB/day**. All customers have timeseries data, so we will **never change or delete** any of it. Is this a good solution for this? What my concern is that the **master nodes** are **useless** and bored. I can't see why they would ever have to work with more than **5%** of their power but still they would need **30%** of the **budget**. **At what point do master nodes become useful**? Shard allocation and cluster managment doesn't sound resource intensive. ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5ef931a1f14d181529d718e3f97df07fb14b2ab9.png) **Concept 3** , Hot-Warm: Is the following concept better for timeseries data? The data is still around 20GB/day. I am more happy with this concept because the **master nodes** are **also the warm nodes.** I don't think the warm nodes will get a lot of traffic so they should be fine. Or is this a bad idea? ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/5/85a648dedbbc9bea0b7008e253c975e00804af24.png) Is the following concept really better? It would add a lot of costs. (If there is a good reason for master nodes we are happy to pay for it, but not if they are not that important) ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/0/b0171234c75500a39b55bf258c9600bbd271b1c7.png) **Concept 4** , Hot-Warm-Cold-Master: This is for the biggest project. **200GB/day** or more. I think in the following concept coordination nodes are missing. They sound really important when having such a big cluster and so much data. What is your opinion on **coordination nodes**? What configuration should they have? ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/5/a58e71e907efc2a905e07a952ef1496256f18014.png) To **sum up** some questions: - How important are master nodes, at what point should we use them? - What is a good cluster concept for timeseries data? - Is there anything "stupid" in my concepts? Open for improvements. - What do your clusters look like? For comparison. Thank you for your time, _defalt_ ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2020, 8:41am UTC](https://discuss.elastic.co/t/best-cluster-concept-timeseries-data-and-more/241482/2 "2020-07-17T08:41:54Z")
When sizing Elasticsearch you will need to take retention period into account. The total data size will drive the number of nodes required in the cluster and determine the optimal cluster structure. ---
**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\ **Post date:** [July 17, 2020, 8:49am UTC](https://discuss.elastic.co/t/best-cluster-concept-timeseries-data-and-more/241482/3 "2020-07-17T08:49:48Z")
Yes, and I know that. So I added the data size in all my questions. For the retention period we are really flexible and I think we will only need ILM for the bigger clusters. We will scale the hot nodes according to the amount of data and they will keep the data for 1-5Months I guess. What do you think of the concepts I showed above according to the data size? - 200MB/day. - 20GB/day. - 200GB/day. Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2020, 9:27am UTC](https://discuss.elastic.co/t/best-cluster-concept-timeseries-data-and-more/241482/4 "2020-07-17T09:27:43Z")
20GB a day is not a lot of data so I would expect a simple 3-node cluster where all nodes hold data and are master eligible to handle that. There should be no need for dedicated master nodes unless you need to add nodes to get more storage. For larger volumes it often depends on the relation between indexed volume and retention, e.g. how fast your data turn over. ---
**Author:** ![defalt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/defalt/32/71379_2.png) [@defalt](https://discuss.elastic.co/u/defalt)\ **Post date:** [July 17, 2020, 10:07am UTC](https://discuss.elastic.co/t/best-cluster-concept-timeseries-data-and-more/241482/5 "2020-07-17T10:07:04Z")
Ok so for the biggest cluster of 200GB/day we would have a retention of 20-50 days. After that they should be transferred to warm nodes. So we would have 3-4 Hot nodes with 2TB SDD and more warm nodes in the back. Is this a good idea or should we just use a Master/Data structure for this? > [@Christian\_Dahlqvist](#): > > There should be no need for dedicated master nodes unless you need to add nodes to get more storage. Lets say after 2 years we need more storage. Would we need to buy 3 new nodes to be master eligible (failproof) and new nodes for storage? Lets say we have 3 Nodes and they are full, what would be the next step? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 14, 2020, 10:07am UTC](https://discuss.elastic.co/t/best-cluster-concept-timeseries-data-and-more/241482/6 "2020-08-14T10:07:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.