# Scaling up **URL:** \ **Category:** Elasticsearch\ **Created:** [April 25, 2012, 8:20am UTC](https://discuss.elastic.co/t/scaling-up/7462 "2012-04-25T08:20:04Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![anghelutar](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@anghelutar](https://discuss.elastic.co/u/anghelutar)\ **Post date:** [April 25, 2012, 8:20am UTC](https://discuss.elastic.co/t/scaling-up/7462/1 "2012-04-25T08:20:04Z")
Hi everybody, We have a fairly big cluster, which keeps growing every day. My question is very simple: how do we know when to add a new node to the cluster? Thank you, roxana ---
**Author:** ![anghelutar](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@anghelutar](https://discuss.elastic.co/u/anghelutar)\ **Post date:** [April 25, 2012, 8:51am UTC](https://discuss.elastic.co/t/scaling-up/7462/2 "2012-04-25T08:51:49Z")
I realize the question is a bit vague. To elaborate more: we have multiple indexes growing, but we also have new indexes added up to the cluster. Does anyone have experience with this? roxana On Apr 25, 10:20 am, anghelutar [anghelu...@gmail.com](mailto:anghelu...@gmail.com) wrote: > Hi everybody, > > We have a fairly big cluster, which keeps growing every day. > My question is very simple: how do we know when to add a new node to > the cluster? > > Thank you, > roxana ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [April 26, 2012, 3:35am UTC](https://discuss.elastic.co/t/scaling-up/7462/3 "2012-04-26T03:35:49Z")
Roxana, How do you know when to add a new node was the original question. Here are some signs: - When performance (e.g. query latency or throughput) starts to suffer - When you don't have enough disk space - When disk IO on existing nodes is at 100% and is slowing down indexing or searching - When shard(s) on a given node become too big for that node and you start running out of memory You can see pretty much all these signs using a tool like SPM for ES: [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) HTH, Otis On Wednesday, April 25, 2012 4:51:49 AM UTC-4, anghelutar wrote: > I realize the question is a bit vague. To elaborate more: we have > multiple indexes growing, but we also have new indexes added up to the > cluster. > > Does anyone have experience with this? > > roxana > > On Apr 25, 10:20 am, anghelutar [anghelu...@gmail.com](mailto:anghelu...@gmail.com) wrote: > > > Hi everybody, > > > > We have a fairly big cluster, which keeps growing every day. > > My question is very simple: how do we know when to add a new node to > > the cluster? > > > > Thank you, > > roxana ---
**Author:** ![anghelutar](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@anghelutar](https://discuss.elastic.co/u/anghelutar)\ **Post date:** [April 26, 2012, 11:17am UTC](https://discuss.elastic.co/t/scaling-up/7462/4 "2012-04-26T11:17:52Z")
Thank you, Otis! I was wondering actually whether it's possible to anticipate any of these signs and to have an alerting system beforehand. The reason is that it takes time to rebalance the cluster, during which things don't work out smoothly. I am specifically interested in avoiding the OOM errors, is there a formula to estimate when we will hit them, function of the types of queries, number of nodes, number of indexes and number of shards per index? Finally, what would be the best architecture for a system with relatively few searches, but quite complicated? roxana On Apr 26, 5:35 am, Otis Gospodnetic [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote: > Roxana, > > How do you know when to add a new node was the original question. > > Here are some signs: > > - When performance (e.g. query latency or throughput) starts to suffer > - When you don't have enough disk space > - When disk IO on existing nodes is at 100% and is slowing down indexing or > searching > - When shard(s) on a given node become too big for that node and you start > running out of memory > > You can see pretty much all these signs using a tool like SPM for > ES:[Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > HTH, > Otis > > On Wednesday, April 25, 2012 4:51:49 AM UTC-4, anghelutar wrote: > > > I realize the question is a bit vague. To elaborate more: we have > > multiple indexes growing, but we also have new indexes added up to the > > cluster. > > > Does anyone have experience with this? > > > roxana > > > On Apr 25, 10:20 am, anghelutar [anghelu...@gmail.com](mailto:anghelu...@gmail.com) wrote: > > > > > Hi everybody, > > > > We have a fairly big cluster, which keeps growing every day. > > > My question is very simple: how do we know when to add a new node to > > > the cluster? > > > > Thank you, > > > roxana ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [April 26, 2012, 8:36pm UTC](https://discuss.elastic.co/t/scaling-up/7462/5 "2012-04-26T20:36:00Z")
Hello, On Thursday, April 26, 2012 7:17:52 AM UTC-4, anghelutar wrote: > Thank you, Otis! > > I was wondering actually whether it's possible to anticipate any of > these signs and to have an alerting system beforehand. The reason is > that it takes time to rebalance the cluster, during which things don't > work out smoothly. It is. Have a look at SPM for ES. Alerts are baked in, though currently hidden in the UI. But that would allow you to set various alert rules/thresholds and be notified when those thresholds are reached. This would in turn be the sign to start thinking about expansion. > I am specifically interested in avoiding the OOM errors, is there a > formula to estimate when we will hit them, function of the types of > queries, number of nodes, number of indexes and number > of shards per index? There is nothing that I know of that is actually accurate. Lots of variables. > Finally, what would be the best architecture for a system with > relatively few searches, but quite complicated? Uh, that's hard to answer without knowing the details. The only thing I could say with certainty is that you wouldn't need many replicas because of the low query load and that you would probably want small shards if queries are really complex to maximize parallelization and minimize latency of individual shard queries. Otis > On Apr 26, 5:35 am, Otis Gospodnetic [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) > wrote: > > > Roxana, > > > > How do you know when to add a new node was the original question. > > > > Here are some signs: > > > > - When performance (e.g. query latency or throughput) starts to suffer > > - When you don't have enough disk space > > - When disk IO on existing nodes is at 100% and is slowing down indexing > > or > > searching > > - When shard(s) on a given node become too big for that node and you > > start > > running out of memory > > > > You can see pretty much all these signs using a tool like SPM for > > ES:[Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > > > HTH, > > Otis > > > > On Wednesday, April 25, 2012 4:51:49 AM UTC-4, anghelutar wrote: > > > > > I realize the question is a bit vague. To elaborate more: we have > > > multiple indexes growing, but we also have new indexes added up to the > > > cluster. > > > > > Does anyone have experience with this? > > > > > roxana > > > > > On Apr 25, 10:20 am, anghelutar [anghelu...@gmail.com](mailto:anghelu...@gmail.com) wrote: > > > > > > > Hi everybody, > > > > > > We have a fairly big cluster, which keeps growing every day. > > > > My question is very simple: how do we know when to add a new node to > > > > the cluster? > > > > > > Thank you, > > > > roxana ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 29, 2012, 4:30pm UTC](https://discuss.elastic.co/t/scaling-up/7462/6 "2012-04-29T16:30:57Z")
Its not easy to estimate requires size, but the first thing you want to keep an eye on is JVM heap usage (which will cause OOM). Since you have a slowly evolving system, you can monitor it using the node stats API, and if it reached ~85-90% of the memory and keep at it for an hour or so, its time to add a node. You will have enough buffer time for relocation to happen while the system is still ok. On Thu, Apr 26, 2012 at 2:17 PM, anghelutar [anghelutar@gmail.com](mailto:anghelutar@gmail.com) wrote: > Thank you, Otis! > > I was wondering actually whether it's possible to anticipate any of > these signs and to have an alerting system beforehand. The reason is > that it takes time to rebalance the cluster, during which things don't > work out smoothly. > > I am specifically interested in avoiding the OOM errors, is there a > formula to estimate when we will hit them, function of the types of > queries, number of nodes, number of indexes and number > of shards per index? > > Finally, what would be the best architecture for a system with > relatively few searches, but quite complicated? > > roxana > > On Apr 26, 5:35 am, Otis Gospodnetic [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) > wrote: > > > Roxana, > > > > How do you know when to add a new node was the original question. > > > > Here are some signs: > > > > - When performance (e.g. query latency or throughput) starts to suffer > > - When you don't have enough disk space > > - When disk IO on existing nodes is at 100% and is slowing down indexing > > or > > searching > > - When shard(s) on a given node become too big for that node and you > > start > > running out of memory > > > > You can see pretty much all these signs using a tool like SPM for > > ES:[Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > > > HTH, > > Otis > > > > On Wednesday, April 25, 2012 4:51:49 AM UTC-4, anghelutar wrote: > > > > > I realize the question is a bit vague. To elaborate more: we have > > > multiple indexes growing, but we also have new indexes added up to the > > > cluster. > > > > > Does anyone have experience with this? > > > > > roxana > > > > > On Apr 25, 10:20 am, anghelutar [anghelu...@gmail.com](mailto:anghelu...@gmail.com) wrote: > > > > > > > Hi everybody, > > > > > > We have a fairly big cluster, which keeps growing every day. > > > > My question is very simple: how do we know when to add a new node to > > > > the cluster? > > > > > > Thank you, > > > > roxana ---
**Author:** ![anghelutar](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@anghelutar](https://discuss.elastic.co/u/anghelutar)\ **Post date:** [April 30, 2012, 10:22pm UTC](https://discuss.elastic.co/t/scaling-up/7462/7 "2012-04-30T22:22:47Z")
When issuing a query, is there any way to tell ES to limit searches to only 1 server for each shard, so that the memory is consumed only on that server? (so basically the other server is used only when the first one is down) Thanks a lot, roxana On Apr 29, 6:30 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Its not easy to estimate requires size, but the first thing you want to > keep an eye on is JVM heap usage (which will cause OOM). Since you have a > slowly evolving system, you can monitor it using the node stats API, and if > it reached ~85-90% of the memory and keep at it for an hour or so, its time > to add a node. You will have enough buffer time for relocation to happen > while the system is still ok. > > On Thu, Apr 26, 2012 at 2:17 PM, anghelutar [anghelu...@gmail.com](mailto:anghelu...@gmail.com) wrote: > > > Thank you, Otis! > > > I was wondering actually whether it's possible to anticipate any of > > these signs and to have an alerting system beforehand. The reason is > > that it takes time to rebalance the cluster, during which things don't > > work out smoothly. > > > I am specifically interested in avoiding the OOM errors, is there a > > formula to estimate when we will hit them, function of the types of > > queries, number of nodes, number of indexes and number > > of shards per index? > > > Finally, what would be the best architecture for a system with > > relatively few searches, but quite complicated? > > > roxana > > > On Apr 26, 5:35 am, Otis Gospodnetic [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) > > wrote: > > > > > Roxana, > > > > How do you know when to add a new node was the original question. > > > > Here are some signs: > > > > - When performance (e.g. query latency or throughput) starts to suffer > > > - When you don't have enough disk space > > > - When disk IO on existing nodes is at 100% and is slowing down indexing > > > or > > > searching > > > - When shard(s) on a given node become too big for that node and you > > > start > > > running out of memory > > > > You can see pretty much all these signs using a tool like SPM for > > > ES:[Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > > > HTH, > > > Otis > > > > On Wednesday, April 25, 2012 4:51:49 AM UTC-4, anghelutar wrote: > > > > > I realize the question is a bit vague. To elaborate more: we have > > > > multiple indexes growing, but we also have new indexes added up to the > > > > cluster. > > > > > Does anyone have experience with this? > > > > > roxana > > > > > On Apr 25, 10:20 am, anghelutar [anghelu...@gmail.com](mailto:anghelu...@gmail.com) wrote: > > > > > > > > > Hi everybody, > > > > > > We have a fairly big cluster, which keeps growing every day. > > > > > My question is very simple: how do we know when to add a new node to > > > > > the cluster? > > > > > > Thank you, > > > > > roxana ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 1, 2012, 3:03pm UTC](https://discuss.elastic.co/t/scaling-up/7462/8 "2012-05-01T15:03:58Z")
You could set the preference in the search request to "\_primary": [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/preference.html). On Tue, May 1, 2012 at 1:22 AM, anghelutar [anghelutar@gmail.com](mailto:anghelutar@gmail.com) wrote: > When issuing a query, is there any way to tell ES to limit searches to > only 1 server for each shard, so that the memory is consumed only on > that server? (so basically the other server is used only when the > first one is down) > > Thanks a lot, > roxana > > On Apr 29, 6:30 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Its not easy to estimate requires size, but the first thing you want to > > keep an eye on is JVM heap usage (which will cause OOM). Since you have a > > slowly evolving system, you can monitor it using the node stats API, and > > if > > it reached ~85-90% of the memory and keep at it for an hour or so, its > > time > > to add a node. You will have enough buffer time for relocation to happen > > while the system is still ok. > > > > On Thu, Apr 26, 2012 at 2:17 PM, anghelutar [anghelu...@gmail.com](mailto:anghelu...@gmail.com) > > wrote: > > > > > Thank you, Otis! > > > > > I was wondering actually whether it's possible to anticipate any of > > > these signs and to have an alerting system beforehand. The reason is > > > that it takes time to rebalance the cluster, during which things don't > > > work out smoothly. > > > > > I am specifically interested in avoiding the OOM errors, is there a > > > formula to estimate when we will hit them, function of the types of > > > queries, number of nodes, number of indexes and number > > > of shards per index? > > > > > Finally, what would be the best architecture for a system with > > > relatively few searches, but quite complicated? > > > > > roxana > > > > > On Apr 26, 5:35 am, Otis Gospodnetic [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) > > > wrote: > > > > > > > Roxana, > > > > > > How do you know when to add a new node was the original question. > > > > > > Here are some signs: > > > > > > - When performance (e.g. query latency or throughput) starts to > > > > suffer > > > > - When you don't have enough disk space > > > > - When disk IO on existing nodes is at 100% and is slowing down > > > > indexing > > > > or > > > > searching > > > > - When shard(s) on a given node become too big for that node and you > > > > start > > > > running out of memory > > > > > > You can see pretty much all these signs using a tool like SPM for > > > > ES:[Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > > > > > HTH, > > > > Otis > > > > > > On Wednesday, April 25, 2012 4:51:49 AM UTC-4, anghelutar wrote: > > > > > > > I realize the question is a bit vague. To elaborate more: we have > > > > > multiple indexes growing, but we also have new indexes added up to > > > > > the > > > > > cluster. > > > > > > > Does anyone have experience with this? > > > > > > > roxana > > > > > > > On Apr 25, 10:20 am, anghelutar [anghelu...@gmail.com](mailto:anghelu...@gmail.com) wrote: > > > > > > > > > > > Hi everybody, > > > > > > > > We have a fairly big cluster, which keeps growing every day. > > > > > > My question is very simple: how do we know when to add a new > > > > > > node to > > > > > > the cluster? > > > > > > > > Thank you, > > > > > > roxana ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:30am UTC](https://discuss.elastic.co/t/scaling-up/7462/9 "2017-07-06T03:30:25Z")