# Heterogeneous clusters **URL:** \ **Category:** Elasticsearch\ **Created:** [September 17, 2012, 8:09pm UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044 "2012-09-17T20:09:59Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 17, 2012, 8:09pm UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/1 "2012-09-17T20:09:59Z")
Hi there, We're working on a project that is going to use ES on a pretty large scale. However we're thinking about deploying it on a lot of small boxes that also run other tasks. The idea is to keep the overhead per server minimal, and require no other servers. The amount of data indexed can however get quite much: rough estimate might be 100GB / node, with a sliding window (e.g. drop indices older then X days). In practice this means that it runs on a box with 64GB RAM and SSD's, a 16GB RAM with 12x 1TB and also on a 1.7GB RAM with 300GB NFS storage. Does ES handle sharding regarding the resources or do we have a "problem"? Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. -- ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [September 18, 2012, 9:31am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/2 "2012-09-18T09:31:23Z")
> In practice this means that it runs on a box with 64GB RAM and SSD's, > a 16GB RAM with 12x 1TB and also on a 1.7GB RAM with 300GB NFS > storage. Does ES handle sharding regarding the resources or do we have > a "problem"? Houston calling 😉 ES treats all nodes as equal, so you really want to make your cluster as homogeneous as possible clint -- ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 18, 2012, 9:45am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/3 "2012-09-18T09:45:14Z")
OK, sounds like we have a problem 😉 Random idea: I see that ES can run on 1GB ram. Does it make any sense to run 8 instances on a large box, and just one on a small box? Or am I trying to do something that's just not designed to be like this? Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/9/18 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > > In practice this means that it runs on a box with 64GB RAM and SSD's, > > a 16GB RAM with 12x 1TB and also on a 1.7GB RAM with 300GB NFS > > storage. Does ES handle sharding regarding the resources or do we have > > a "problem"? > > Houston calling 😉 > > ES treats all nodes as equal, so you really want to make your cluster as > homogeneous as possible > > clint > > -- -- ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [September 18, 2012, 9:52am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/4 "2012-09-18T09:52:41Z")
On Tue, 2012-09-18 at 11:45 +0200, Robin Verlangen wrote: > OK, sounds like we have a problem 😉 > > Random idea: I see that ES can run on 1GB ram. Does it make any sense > to run 8 instances on a large box, and just one on a small box? Or am > I trying to do something that's just not designed to be like this? The problem you have with that is that you might end up with primaries and replicas on one box. That box goes down, and you've lost your data. clint > Best regards, > > Robin Verlangen > Software engineer > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > E robin@us2.nl > > Disclaimer: The information contained in this message and attachments > is intended solely for the attention and use of the named addressee > and may be confidential. If you are not the intended recipient, you > are reminded that the information remains the property of the sender. > You must not use, disclose, distribute, copy, print or rely on this > e-mail. If you have received this message in error, please contact the > sender immediately and irrevocably delete this message and any copies. > > 2012/9/18 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > > ``` > > > > In practice this means that it runs on a box with 64GB RAM > and SSD's, > > a 16GB RAM with 12x 1TB and also on a 1.7GB RAM with 300GB > NFS > > storage. Does ES handle sharding regarding the resources or > do we have > > a "problem"? > > > Houston calling ;) > > ES treats all nodes as equal, so you really want to make your > cluster as > homogeneous as possible > > clint > > > -- > > ``` > > -- -- ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [September 18, 2012, 9:53am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/5 "2012-09-18T09:53:53Z")
> The problem you have with that is that you might end up with primaries > and replicas on one box. That box goes down, and you've lost your data. Also, if you're only running with 1GB a proportionately larger amount of RAM will be used for java, code, and state than you'd have with bigger boxes with more RAM clint -- ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 18, 2012, 9:55am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/6 "2012-09-18T09:55:14Z")
That makes sense indeed. I'll go through our design again and see how we can resolve this without adding a lot of overhead on machines. Do you know whether this heterogeneous support is something in line for the near-future? Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/9/18 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > > The problem you have with that is that you might end up with primaries > > and replicas on one box. That box goes down, and you've lost your data. > > Also, if you're only running with 1GB a proportionately larger amount of > RAM will be used for java, code, and state than you'd have with bigger > boxes with more RAM > > clint > > -- -- ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 18, 2012, 9:56am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/7 "2012-09-18T09:56:04Z")
There is a setting called: cluster.routing.allocation.same\_shard.host, where if you set it to true, it will make sure not to allocate a shard and a replica on the same "host", regardless of instances running on the host (where host is based on the network address). On Sep 18, 2012, at 11:52 AM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > On Tue, 2012-09-18 at 11:45 +0200, Robin Verlangen wrote: > > > OK, sounds like we have a problem 😉 > > > > Random idea: I see that ES can run on 1GB ram. Does it make any sense > > to run 8 instances on a large box, and just one on a small box? Or am > > I trying to do something that's just not designed to be like this? > > The problem you have with that is that you might end up with primaries > and replicas on one box. That box goes down, and you've lost your data. > > clint > > > Best regards, > > > > Robin Verlangen > > Software engineer > > > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > E robin@us2.nl > > > > Disclaimer: The information contained in this message and attachments > > is intended solely for the attention and use of the named addressee > > and may be confidential. If you are not the intended recipient, you > > are reminded that the information remains the property of the sender. > > You must not use, disclose, distribute, copy, print or rely on this > > e-mail. If you have received this message in error, please contact the > > sender immediately and irrevocably delete this message and any copies. > > > > 2012/9/18 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > > > > > In practice this means that it runs on a box with 64GB RAM > > > and SSD's, > > > a 16GB RAM with 12x 1TB and also on a 1.7GB RAM with 300GB > > > NFS > > > storage. Does ES handle sharding regarding the resources or > > > do we have > > > a "problem"? > > > > ``` > > Houston calling ;) > > > > ES treats all nodes as equal, so you really want to make your > > cluster as > > homogeneous as possible > > > > clint > > > > -- > > > > ``` > > > > -- > > -- -- ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 18, 2012, 10:01am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/8 "2012-09-18T10:01:58Z")
That sounds like a possible workaround without real problems. However I think ES is not designed for this purpose, we might need to reconsider the pros and cons. Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/9/18 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) > There is a setting called: cluster.routing.allocation.same\_shard.host, > where if you set it to true, it will make sure not to allocate a shard and > a replica on the same "host", regardless of instances running on the host > (where host is based on the network address). > > On Sep 18, 2012, at 11:52 AM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > wrote: > > > On Tue, 2012-09-18 at 11:45 +0200, Robin Verlangen wrote: > > > > > OK, sounds like we have a problem 😉 > > > > > > Random idea: I see that ES can run on 1GB ram. Does it make any sense > > > to run 8 instances on a large box, and just one on a small box? Or am > > > I trying to do something that's just not designed to be like this? > > > > The problem you have with that is that you might end up with primaries > > and replicas on one box. That box goes down, and you've lost your data. > > > > clint > > > > > Best regards, > > > > > > Robin Verlangen > > > Software engineer > > > > > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > > E robin@us2.nl > > > > > > Disclaimer: The information contained in this message and attachments > > > is intended solely for the attention and use of the named addressee > > > and may be confidential. If you are not the intended recipient, you > > > are reminded that the information remains the property of the sender. > > > You must not use, disclose, distribute, copy, print or rely on this > > > e-mail. If you have received this message in error, please contact the > > > sender immediately and irrevocably delete this message and any copies. > > > > > > 2012/9/18 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > > > > > > > In practice this means that it runs on a box with 64GB RAM > > > > and SSD's, > > > > a 16GB RAM with 12x 1TB and also on a 1.7GB RAM with 300GB > > > > NFS > > > > storage. Does ES handle sharding regarding the resources or > > > > do we have > > > > a "problem"? > > > > > > ``` > > > Houston calling ;) > > > > > > ES treats all nodes as equal, so you really want to make your > > > cluster as > > > homogeneous as possible > > > > > > clint > > > > > > -- > > > > > > ``` > > > > > > -- > > > > -- > > -- -- ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 18, 2012, 10:11am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/9 "2012-09-18T10:11:49Z")
Agreed, you can try and work around it by having several instances running and so on, but it makes more sense to have same size nodes in the cluster. On Sep 18, 2012, at 12:01 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote: > That sounds like a possible workaround without real problems. However I think ES is not designed for this purpose, we might need to reconsider the pros and cons. > > Best regards, > > Robin Verlangen > Software engineer > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > E robin@us2.nl > > Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. > > 2012/9/18 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) > There is a setting called: cluster.routing.allocation.same\_shard.host, where if you set it to true, it will make sure not to allocate a shard and a replica on the same "host", regardless of instances running on the host (where host is based on the network address). > > On Sep 18, 2012, at 11:52 AM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote: > > > On Tue, 2012-09-18 at 11:45 +0200, Robin Verlangen wrote: > > > > > OK, sounds like we have a problem 😉 > > > > > > Random idea: I see that ES can run on 1GB ram. Does it make any sense > > > to run 8 instances on a large box, and just one on a small box? Or am > > > I trying to do something that's just not designed to be like this? > > > > The problem you have with that is that you might end up with primaries > > and replicas on one box. That box goes down, and you've lost your data. > > > > clint > > > > > Best regards, > > > > > > Robin Verlangen > > > Software engineer > > > > > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > > E robin@us2.nl > > > > > > Disclaimer: The information contained in this message and attachments > > > is intended solely for the attention and use of the named addressee > > > and may be confidential. If you are not the intended recipient, you > > > are reminded that the information remains the property of the sender. > > > You must not use, disclose, distribute, copy, print or rely on this > > > e-mail. If you have received this message in error, please contact the > > > sender immediately and irrevocably delete this message and any copies. > > > > > > 2012/9/18 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > > > > > > > In practice this means that it runs on a box with 64GB RAM > > > > and SSD's, > > > > a 16GB RAM with 12x 1TB and also on a 1.7GB RAM with 300GB > > > > NFS > > > > storage. Does ES handle sharding regarding the resources or > > > > do we have > > > > a "problem"? > > > > > > ``` > > > Houston calling ;) > > > > > > ES treats all nodes as equal, so you really want to make your > > > cluster as > > > homogeneous as possible > > > > > > clint > > > > > > -- > > > > > > ``` > > > > > > -- > > > > -- > > -- > > -- -- ---
**Author:** ![Robin\_Verlangen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robin_verlangen/32/1542_2.png) [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)\ **Post date:** [September 18, 2012, 10:13am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/10 "2012-09-18T10:13:17Z")
Well actually we would like to spread the load over every machine running our software, but with this ES behavior that doesn't sound like a good plan. I think we should go with an option of online running ES on a subset of machines that are equal in terms of resources. Best regards, Robin Verlangen _Software engineer_ \* \* W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) E robin@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. 2012/9/18 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) > Agreed, you can try and work around it by having several instances running > and so on, but it makes more sense to have same size nodes in the cluster. > > On Sep 18, 2012, at 12:01 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote: > > That sounds like a possible workaround without real problems. However I > think ES is not designed for this purpose, we might need to reconsider the > pros and cons. > > Best regards, > > Robin Verlangen > _Software engineer_ > \* > \* > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > E robin@us2.nl > > Disclaimer: The information contained in this message and attachments is > intended solely for the attention and use of the named addressee and may be > confidential. If you are not the intended recipient, you are reminded that > the information remains the property of the sender. You must not use, > disclose, distribute, copy, print or rely on this e-mail. If you have > received this message in error, please contact the sender immediately and > irrevocably delete this message and any copies. > > 2012/9/18 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) > > > There is a setting called: cluster.routing.allocation.same\_shard.host, > > where if you set it to true, it will make sure not to allocate a shard and > > a replica on the same "host", regardless of instances running on the host > > (where host is based on the network address). > > > > On Sep 18, 2012, at 11:52 AM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > > wrote: > > > > > On Tue, 2012-09-18 at 11:45 +0200, Robin Verlangen wrote: > > > > > > > OK, sounds like we have a problem 😉 > > > > > > > > Random idea: I see that ES can run on 1GB ram. Does it make any sense > > > > to run 8 instances on a large box, and just one on a small box? Or am > > > > I trying to do something that's just not designed to be like this? > > > > > > The problem you have with that is that you might end up with primaries > > > and replicas on one box. That box goes down, and you've lost your data. > > > > > > clint > > > > > > > Best regards, > > > > > > > > Robin Verlangen > > > > Software engineer > > > > > > > > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl) > > > > E robin@us2.nl > > > > > > > > Disclaimer: The information contained in this message and attachments > > > > is intended solely for the attention and use of the named addressee > > > > and may be confidential. If you are not the intended recipient, you > > > > are reminded that the information remains the property of the sender. > > > > You must not use, disclose, distribute, copy, print or rely on this > > > > e-mail. If you have received this message in error, please contact the > > > > sender immediately and irrevocably delete this message and any copies. > > > > > > > > 2012/9/18 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) > > > > > > > > > In practice this means that it runs on a box with 64GB RAM > > > > > and SSD's, > > > > > a 16GB RAM with 12x 1TB and also on a 1.7GB RAM with 300GB > > > > > NFS > > > > > storage. Does ES handle sharding regarding the resources or > > > > > do we have > > > > > a "problem"? > > > > > > > > ``` > > > > Houston calling ;) > > > > > > > > ES treats all nodes as equal, so you really want to make your > > > > cluster as > > > > homogeneous as possible > > > > > > > > clint > > > > > > > > -- > > > > > > > > ``` > > > > > > > > -- > > > > > > -- > > > > -- > > -- > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:12am UTC](https://discuss.elastic.co/t/heterogeneous-clusters/9044/11 "2017-07-06T03:12:30Z")