# Best cluster environment for search **URL:** \ **Category:** Elasticsearch\ **Created:** [June 2, 2014, 7:52pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866 "2014-06-02T19:52:12Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Marcelo\_Paes\_Rech](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@Marcelo\_Paes\_Rech](https://discuss.elastic.co/u/Marcelo_Paes_Rech)\ **Post date:** [June 2, 2014, 7:52pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/1 "2014-06-02T19:52:12Z")
Hi guys, I'm looking for an article or a guide for the best cluster configuration. I read a lot of articles like "change this configuration" and "you must create X shards per node" but I didn't saw nothing like ElasticSearch Official guide for creating a cluster. What I would like to know are informations like. - How to calculate how many shards will be good for the cluster. - How many shards do we need per node? And if this is variable, how do I calculate this? - How much memory do I need per node and how many nodes? I think ElasticSearch is well documentated. But it is very fragmented. Best regards. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b8894495-e64b-4796-9eb4-e49e1b9ce556%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b8894495-e64b-4796-9eb4-e49e1b9ce556%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [June 2, 2014, 8:08pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/2 "2014-06-02T20:08:39Z")
On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech \<[marcelopaesrech@gmail.com](mailto:marcelopaesrech@gmail.com) > wrote: Hi guys, > I'm looking for an article or a guide for the best cluster configuration. > I read a lot of articles like "change this configuration" and "you must > create X shards per node" but I didn't saw nothing like Elasticsearch > Official guide for creating a cluster. > > What I would like to know are informations like. > > - How to calculate how many shards will be good for the cluster. > - How many shards do we need per node? And if this is variable, how do I > calculate this? > - How much memory do I need per node and how many nodes? > > I think Elasticsearch is well documentated. But it is very fragmented. For some of these that is because "it depends" is the answer. For example, you'll want larger heaps for aggregations and faceting. There are some rules of thumb: 1. Set Elasticsearch's heap memory to 1/2 of ram but not more then 30GB. Bigger then that and the JVM can't do pointer compression and you effectively lose ram. 2. #1 implies that having much more then 60GB of ram on each node doesn't make a big difference. It helps but its not really as good as having more nodes. 3. The most efficient efficient way of sharding is likely one shard on each node. So if you have 9 nodes and a replication factor of 2 (so 3 total copies) then 3 shards is likely to be more efficient then having 2 or 4. But this only really matters when those shards get lots of traffic. And it breaks down a bit when you get lots of nodes. And the in presence of routing. Its complicated. But these are really just starting points, safe-ish defaults. Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd11f\_mV88p1z2F%2Bm543FEcZkH5va%3DSdF-T0vFoNwuszng%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd11f_mV88p1z2F%2Bm543FEcZkH5va%3DSdF-T0vFoNwuszng%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marcelo\_Paes\_Rech](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@Marcelo\_Paes\_Rech](https://discuss.elastic.co/u/Marcelo_Paes_Rech)\ **Post date:** [June 3, 2014, 10:18pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/3 "2014-06-03T22:18:07Z")
Thanks for your reply Nikolas. It helps a lot. And about the quantity of documents of each shard, or size of each shard. And the need of no data nodes or only master nodes. When is it necessary? Some tests I did, when I increased request's number (like 100 users at same moment, and redo it again and again), 5 nodes with 1 shard and 2 replicas each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The response time start to increase more than 5s (I think less than 1s, in this case, would be acceptable) . This test has a lot of documents (something like 14 millions). Thanks. Regards. Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas Everett escreveu: > On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech \<[marcelo...@gmail.com](mailto:marcelo...@gmail.com) > \\> wrote: > > Hi guys, > > > I'm looking for an article or a guide for the best cluster configuration. > > I read a lot of articles like "change this configuration" and "you must > > create X shards per node" but I didn't saw nothing like Elasticsearch > > Official guide for creating a cluster. > > > > What I would like to know are informations like. > > > > - How to calculate how many shards will be good for the cluster. > > - How many shards do we need per node? And if this is variable, how do I > > calculate this? > > - How much memory do I need per node and how many nodes? > > > > I think Elasticsearch is well documentated. But it is very fragmented. > > For some of these that is because "it depends" is the answer. For > example, you'll want larger heaps for aggregations and faceting. > > There are some rules of thumb: > > 1. Set Elasticsearch's heap memory to 1/2 of ram but not more then 30GB. > Bigger then that and the JVM can't do pointer compression and you > effectively lose ram. > 2. #1 implies that having much more then 60GB of ram on each node doesn't > make a big difference. It helps but its not really as good as having more > nodes. > 3. The most efficient efficient way of sharding is likely one shard on > each node. So if you have 9 nodes and a replication factor of 2 (so 3 > total copies) then 3 shards is likely to be more efficient then having 2 or > 4. But this only really matters when those shards get lots of traffic. > And it breaks down a bit when you get lots of nodes. And the in presence > of routing. Its complicated. > > But these are really just starting points, safe-ish defaults. > > Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [June 3, 2014, 11:33pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/4 "2014-06-03T23:33:37Z")
Can you show your test code? You seem to look at the wrong settings - by adjusting node number, shard number, replica number alone, you can not find out the maximum node performance. E.g. concurrency settings, index optimizations, query optimizations, thread pooling, and most of all, fast disk subsystem I/O is important. Jörg On Wed, Jun 4, 2014 at 12:18 AM, Marcelo Paes Rech \< [marcelopaesrech@gmail.com](mailto:marcelopaesrech@gmail.com)\> wrote: > Thanks for your reply Nikolas. It helps a lot. > > And about the quantity of documents of each shard, or size of each shard. > And the need of no data nodes or only master nodes. When is it necessary? > > Some tests I did, when I increased request's number (like 100 users at > same moment, and redo it again and again), 5 nodes with 1 shard and 2 > replicas each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The > response time start to increase more than 5s (I think less than 1s, in > this case, would be acceptable) . > > This test has a lot of documents (something like 14 millions). > > Thanks. Regards. > > Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas Everett > escreveu: > > > On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech [marcelo...@gmail.com](mailto:marcelo...@gmail.com) > > wrote: > > > > Hi guys, > > > > > I'm looking for an article or a guide for the best cluster > > > configuration. I read a lot of articles like "change this configuration" > > > and "you must create X shards per node" but I didn't saw nothing like > > > Elasticsearch Official guide for creating a cluster. > > > > > > What I would like to know are informations like. > > > > > > - How to calculate how many shards will be good for the cluster. > > > - How many shards do we need per node? And if this is variable, how do I > > > calculate this? > > > - How much memory do I need per node and how many nodes? > > > > > > I think Elasticsearch is well documentated. But it is very fragmented. > > > > For some of these that is because "it depends" is the answer. For > > example, you'll want larger heaps for aggregations and faceting. > > > > There are some rules of thumb: > > > > 1. Set Elasticsearch's heap memory to 1/2 of ram but not more then > > 30GB. Bigger then that and the JVM can't do pointer compression and you > > effectively lose ram. > > 2. #1 implies that having much more then 60GB of ram on each node > > doesn't make a big difference. It helps but its not really as good as > > having more nodes. > > 3. The most efficient efficient way of sharding is likely one shard on > > each node. So if you have 9 nodes and a replication factor of 2 (so 3 > > total copies) then 3 shards is likely to be more efficient then having 2 or > > 4. But this only really matters when those shards get lots of traffic. > > And it breaks down a bit when you get lots of nodes. And the in presence > > of routing. Its complicated. > > > > But these are really just starting points, safe-ish defaults. > > > > Nik > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBBwbpCnXPR%3D9r-yTngRbAUcG\_wiqMTN8Hk6XeggPHLw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBBwbpCnXPR%3D9r-yTngRbAUcG_wiqMTN8Hk6XeggPHLw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marcelo\_Paes\_Rech](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@Marcelo\_Paes\_Rech](https://discuss.elastic.co/u/Marcelo_Paes_Rech)\ **Post date:** [June 4, 2014, 12:40pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/5 "2014-06-04T12:40:27Z")
Hi Jörg. Thanks for your reply. Here is my filter. {"filter": { "terms" : { "\_id" : [ "QSxrbEM8TKe5zr8931xBjA", "wj63ghegRwC6qLsWq2chkA", "hYEhDbAqQwSRxhYfvDgFkg", "4bZmPE1fTYqijphRyyWiuQ", "Fhq53yYyT3CEw6vclKu\_NA", "XL2atBraTEyx57MefjFVhA", "951i0dZkT064FlQkzHnnWA", "O8Ixbir1TrGT\_IA3wKfsHg", "8k4U7KsuTmsThqxy-5YaKw", "GNOoQTHglf22kzcE7EOf8g", "-RQeY48fTg2kYnh2M4E1cQ", "u8DGBdfVR9WRVj6d9E4Ebw", "WFHSXd7UQvCMYFBhFcTsng", "qnQ7q7FyTsg397lM1EWgqA", "wRQtUzdMRy2qOkMCNxdpgA", "Ll83iglxSUS\_Gs7mjkMt8w", "d2sxZ1oBTfuvAfov5EJ0iw", "cyht-vB4Q-mMSg9N5jcGXg", "bNSVaO47QTOCkfJhWo0qjg", "BHuhm55IRerKnynJ8WgFTw", "fHKA4PF2QteWm8E7dW7CAw", "DLE6A7tyQJ-zcKcCa6IPSA", "qfelTW7-SuGRQ0GKbngARA", "R7VHHJhYsUqfuxYof8BJ8w", "W4PqiJfPSlSFjVKFsGkA4Q", "Juq62zOsRdheuW3O6Gb2KA", "U9v0IKj\_RrgRNjE31ZTt2g", "uNHa0kOOT5qjPpzxZcs35A", "SwOgVNgIRwyVU3pEEycBuQ", "LaEpxFGIQgCArsNZ2rd4Pw", "CiJ9gouZsbmTtxTWx7w6lA", "TaQV\_I01RfCq3B6uAtIBoQ", "9Jpjo5k-RlGfLVLF6nDgze", "57YpjRdASsrrae-RD3spog", "bmA4EWFSTiKUaDzaNcCFKQ", "Fui9z\_UbRe6AY1VhAr8Crw", "2PORr5BzSDOmBXgmQkO5Zg", "snfwTmtuTv-uj5mOWSJpgA", "0nHIrtePSaeW8aWArh\_Mrg", "s0g9QHnjTgWX3rCIu1g0Hg", "Jl67fACuQvCFgZxXAFtDOg" ], "\_cache" : true, "\_cache\_key" : "my\_terms\_cache" } } } I already used "_ids filter_" but I got same behaviour. One thing that I realized is that one of the cluster's nodes is increasing the Search Thread Pool (something like Queue: 50 and Count: 47) and the others don't (something like Queue: 0 and Count: 1). If I remove this node from the cluster another one starts with the same problem. My current environment is: - 7 Data nodes with 16Gb (8Gb for ES)and 8 cores each one; - 4 Load balancer Nodes (no data, no master) with 4Gb (3Gb for ES) and 8 cores each one; - 4 MasterNodes (only master, no data) with 4Gb (3Gb for ES) and 8 cores each one; - Thread Pool Search 47 (the others are standard config); - 7 Shards and 2 replicas Index; - 14.6Gb Index size (14.524.273 documents); I'm executing this filter with 50 concurrent users. Regards Em terça-feira, 3 de junho de 2014 20h33min45s UTC-3, Jörg Prante escreveu: > Can you show your test code? > > You seem to look at the wrong settings - by adjusting node number, shard > number, replica number alone, you can not find out the maximum node > performance. E.g. concurrency settings, index optimizations, query > optimizations, thread pooling, and most of all, fast disk subsystem I/O is > important. > > Jörg > > On Wed, Jun 4, 2014 at 12:18 AM, Marcelo Paes Rech \<[marcelo...@gmail.com](mailto:marcelo...@gmail.com) > \\> wrote: > > > Thanks for your reply Nikolas. It helps a lot. > > > > And about the quantity of documents of each shard, or size of each shard. > > And the need of no data nodes or only master nodes. When is it necessary? > > > > Some tests I did, when I increased request's number (like 100 users at > > same moment, and redo it again and again), 5 nodes with 1 shard and 2 > > replicas each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The > > response time start to increase more than 5s (I think less than 1s, in > > this case, would be acceptable) . > > > > This test has a lot of documents (something like 14 millions). > > > > Thanks. Regards. > > > > Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas Everett > > escreveu: > > > > > On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech [marcelo...@gmail.com](mailto:marcelo...@gmail.com) > > > wrote: > > > > > > Hi guys, > > > > > > > I'm looking for an article or a guide for the best cluster > > > > configuration. I read a lot of articles like "change this configuration" > > > > and "you must create X shards per node" but I didn't saw nothing like > > > > Elasticsearch Official guide for creating a cluster. > > > > > > > > What I would like to know are informations like. > > > > > > > > - How to calculate how many shards will be good for the cluster. > > > > - How many shards do we need per node? And if this is variable, how do > > > > I calculate this? > > > > - How much memory do I need per node and how many nodes? > > > > > > > > I think Elasticsearch is well documentated. But it is very fragmented. > > > > > > For some of these that is because "it depends" is the answer. For > > > example, you'll want larger heaps for aggregations and faceting. > > > > > > There are some rules of thumb: > > > > > > 1. Set Elasticsearch's heap memory to 1/2 of ram but not more then > > > 30GB. Bigger then that and the JVM can't do pointer compression and you > > > effectively lose ram. > > > 2. #1 implies that having much more then 60GB of ram on each node > > > doesn't make a big difference. It helps but its not really as good as > > > having more nodes. > > > 3. The most efficient efficient way of sharding is likely one shard on > > > each node. So if you have 9 nodes and a replication factor of 2 (so 3 > > > total copies) then 3 shards is likely to be more efficient then having 2 or > > > 4. But this only really matters when those shards get lots of traffic. > > > And it breaks down a bit when you get lots of nodes. And the in presence > > > of routing. Its complicated. > > > > > > But these are really just starting points, safe-ish defaults. > > > > > > Nik > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [June 4, 2014, 10:54pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/6 "2014-06-04T22:54:09Z")
Why do you use terms on \_id field and not the the ids filter? ids filter is more efficient since it reuses the \_uid field which is cached by default. Do the terms in the query vary from query to query? If so, caching might kill your heap. Another possible issue is that your query is not distributed to all shards, if the query does not vary from user to user in your test. If so, you created a "hot spot", all the load from the 100 users wold go to a limited number of node with a limited shard count. The search thread pool seems small with 50 searches if you execute searches for 100 users in parallel, this can lead to a congestion of the search module. Why don't you use 100 (at least)? Jörg On Wed, Jun 4, 2014 at 2:40 PM, Marcelo Paes Rech \<[marcelopaesrech@gmail.com](mailto:marcelopaesrech@gmail.com) > wrote: > Hi Jörg. Thanks for your reply. > > Here is my filter. > {"filter": > { > "terms" : { > "\_id" : [ "QSxrbEM8TKe5zr8931xBjA", "wj63ghegRwC6qLsWq2chkA", > "hYEhDbAqQwSRxhYfvDgFkg", "4bZmPE1fTYqijphRyyWiuQ", > "Fhq53yYyT3CEw6vclKu\_NA", "XL2atBraTEyx57MefjFVhA", > "951i0dZkT064FlQkzHnnWA", "O8Ixbir1TrGT\_IA3wKfsHg", > "8k4U7KsuTmsThqxy-5YaKw", "GNOoQTHglf22kzcE7EOf8g", > "-RQeY48fTg2kYnh2M4E1cQ", "u8DGBdfVR9WRVj6d9E4Ebw", > "WFHSXd7UQvCMYFBhFcTsng", "qnQ7q7FyTsg397lM1EWgqA", > "wRQtUzdMRy2qOkMCNxdpgA", "Ll83iglxSUS\_Gs7mjkMt8w", > "d2sxZ1oBTfuvAfov5EJ0iw", "cyht-vB4Q-mMSg9N5jcGXg", > "bNSVaO47QTOCkfJhWo0qjg", "BHuhm55IRerKnynJ8WgFTw", > "fHKA4PF2QteWm8E7dW7CAw", "DLE6A7tyQJ-zcKcCa6IPSA", > "qfelTW7-SuGRQ0GKbngARA", "R7VHHJhYsUqfuxYof8BJ8w", > "W4PqiJfPSlSFjVKFsGkA4Q", "Juq62zOsRdheuW3O6Gb2KA", > "U9v0IKj\_RrgRNjE31ZTt2g", "uNHa0kOOT5qjPpzxZcs35A", > "SwOgVNgIRwyVU3pEEycBuQ", "LaEpxFGIQgCArsNZ2rd4Pw", > "CiJ9gouZsbmTtxTWx7w6lA", "TaQV\_I01RfCq3B6uAtIBoQ", > "9Jpjo5k-RlGfLVLF6nDgze", "57YpjRdASsrrae-RD3spog", > "bmA4EWFSTiKUaDzaNcCFKQ", "Fui9z\_UbRe6AY1VhAr8Crw", > "2PORr5BzSDOmBXgmQkO5Zg", "snfwTmtuTv-uj5mOWSJpgA", > "0nHIrtePSaeW8aWArh\_Mrg", "s0g9QHnjTgWX3rCIu1g0Hg", > "Jl67fACuQvCFgZxXAFtDOg" ], > "\_cache" : true, > "\_cache\_key" : "my\_terms\_cache" > } > } > } > > I already used "_ids filter_" but I got same behaviour. One thing that I > realized is that one of the cluster's nodes is increasing the Search Thread > Pool (something like Queue: 50 and Count: 47) and the others don't > (something like Queue: 0 and Count: 1). If I remove this node from the > cluster another one starts with the same problem. > > My current environment is: > > - 7 Data nodes with 16Gb (8Gb for ES)and 8 cores each one; > - 4 Load balancer Nodes (no data, no master) with 4Gb (3Gb for ES) and 8 > cores each one; > - 4 MasterNodes (only master, no data) with 4Gb (3Gb for ES) and 8 cores > each one; > - Thread Pool Search 47 (the others are standard config); > - 7 Shards and 2 replicas Index; > - 14.6Gb Index size (14.524.273 documents); > > I'm executing this filter with 50 concurrent users. > > Regards > > Em terça-feira, 3 de junho de 2014 20h33min45s UTC-3, Jörg Prante escreveu: > > > Can you show your test code? > > > > You seem to look at the wrong settings - by adjusting node number, shard > > number, replica number alone, you can not find out the maximum node > > performance. E.g. concurrency settings, index optimizations, query > > optimizations, thread pooling, and most of all, fast disk subsystem I/O is > > important. > > > > Jörg > > > > On Wed, Jun 4, 2014 at 12:18 AM, Marcelo Paes Rech [marcelo...@gmail.com](mailto:marcelo...@gmail.com) > > wrote: > > > > > Thanks for your reply Nikolas. It helps a lot. > > > > > > And about the quantity of documents of each shard, or size of each > > > shard. And the need of no data nodes or only master nodes. When is it > > > necessary? > > > > > > Some tests I did, when I increased request's number (like 100 users at > > > same moment, and redo it again and again), 5 nodes with 1 shard and 2 > > > replicas each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The > > > response time start to increase more than 5s (I think less than 1s, in > > > this case, would be acceptable) . > > > > > > This test has a lot of documents (something like 14 millions). > > > > > > Thanks. Regards. > > > > > > Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas Everett > > > escreveu: > > > > > > > On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech \<[marcelo...@gmail.com](mailto:marcelo...@gmail.com) > > > > > > > > > wrote: > > > > > > > > Hi guys, > > > > > > > > > I'm looking for an article or a guide for the best cluster > > > > > configuration. I read a lot of articles like "change this configuration" > > > > > and "you must create X shards per node" but I didn't saw nothing like > > > > > Elasticsearch Official guide for creating a cluster. > > > > > > > > > > What I would like to know are informations like. > > > > > > > > > > - How to calculate how many shards will be good for the cluster. > > > > > - How many shards do we need per node? And if this is variable, how do > > > > > I calculate this? > > > > > - How much memory do I need per node and how many nodes? > > > > > > > > > > I think Elasticsearch is well documentated. But it is very fragmented. > > > > > > > > For some of these that is because "it depends" is the answer. For > > > > example, you'll want larger heaps for aggregations and faceting. > > > > > > > > There are some rules of thumb: > > > > > > > > 1. Set Elasticsearch's heap memory to 1/2 of ram but not more then > > > > 30GB. Bigger then that and the JVM can't do pointer compression and you > > > > effectively lose ram. > > > > 2. #1 implies that having much more then 60GB of ram on each node > > > > doesn't make a big difference. It helps but its not really as good as > > > > having more nodes. > > > > 3. The most efficient efficient way of sharding is likely one shard on > > > > each node. So if you have 9 nodes and a replication factor of 2 (so 3 > > > > total copies) then 3 shards is likely to be more efficient then having 2 or > > > > 4. But this only really matters when those shards get lots of traffic. > > > > And it breaks down a bit when you get lots of nodes. And the in presence > > > > of routing. Its complicated. > > > > > > > > But these are really just starting points, safe-ish defaults. > > > > > > > > Nik > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFC5GxRVG\_k\_C%3D-FBnUXaRx%3DkfHWS\_5srKyjFBKM3\_%2Bg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGFC5GxRVG_k_C%3D-FBnUXaRx%3DkfHWS_5srKyjFBKM3_%2Bg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marcelo\_Paes\_Rech](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@Marcelo\_Paes\_Rech](https://discuss.elastic.co/u/Marcelo_Paes_Rech)\ **Post date:** [June 5, 2014, 12:38pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/7 "2014-06-05T12:38:04Z")
Hi Jörg. Thanks for your reply again. As I said, I already had used ids filter, but I got the same behaviour. I realized what was wrong. Maybe it could be a bug in ES or not. When I executed the filter I included "from" and "size" attibutes. In this case "size" was 999999, but the final result would be just 10 documents. Aparently ES pre-allocates the objects that I say I will use (maybe for performance reasons), but if the final result is not the total (999999), ES doesn't remove remaining pre-allocated objects until the memory (heap) is full. I changed the size attribute to 10 and heap became stable. That's it. Thanks. Regards. Em quarta-feira, 4 de junho de 2014 19h54min15s UTC-3, Jörg Prante escreveu: > Why do you use terms on \_id field and not the the ids filter? ids filter > is more efficient since it reuses the \_uid field which is cached by default. > > Do the terms in the query vary from query to query? If so, caching might > kill your heap. > > Another possible issue is that your query is not distributed to all > shards, if the query does not vary from user to user in your test. If so, > you created a "hot spot", all the load from the 100 users wold go to a > limited number of node with a limited shard count. > > The search thread pool seems small with 50 searches if you execute > searches for 100 users in parallel, this can lead to a congestion of the > search module. Why don't you use 100 (at least)? > > Jörg > > On Wed, Jun 4, 2014 at 2:40 PM, Marcelo Paes Rech \<[marcelo...@gmail.com](mailto:marcelo...@gmail.com) > \\> wrote: > > > Hi Jörg. Thanks for your reply. > > > > Here is my filter. > > {"filter": > > { > > "terms" : { > > "\_id" : [ "QSxrbEM8TKe5zr8931xBjA", "wj63ghegRwC6qLsWq2chkA", > > "hYEhDbAqQwSRxhYfvDgFkg", "4bZmPE1fTYqijphRyyWiuQ", > > "Fhq53yYyT3CEw6vclKu\_NA", "XL2atBraTEyx57MefjFVhA", > > "951i0dZkT064FlQkzHnnWA", "O8Ixbir1TrGT\_IA3wKfsHg", > > "8k4U7KsuTmsThqxy-5YaKw", "GNOoQTHglf22kzcE7EOf8g", > > "-RQeY48fTg2kYnh2M4E1cQ", "u8DGBdfVR9WRVj6d9E4Ebw", > > "WFHSXd7UQvCMYFBhFcTsng", "qnQ7q7FyTsg397lM1EWgqA", > > "wRQtUzdMRy2qOkMCNxdpgA", "Ll83iglxSUS\_Gs7mjkMt8w", > > "d2sxZ1oBTfuvAfov5EJ0iw", "cyht-vB4Q-mMSg9N5jcGXg", > > "bNSVaO47QTOCkfJhWo0qjg", "BHuhm55IRerKnynJ8WgFTw", > > "fHKA4PF2QteWm8E7dW7CAw", "DLE6A7tyQJ-zcKcCa6IPSA", > > "qfelTW7-SuGRQ0GKbngARA", "R7VHHJhYsUqfuxYof8BJ8w", > > "W4PqiJfPSlSFjVKFsGkA4Q", "Juq62zOsRdheuW3O6Gb2KA", > > "U9v0IKj\_RrgRNjE31ZTt2g", "uNHa0kOOT5qjPpzxZcs35A", > > "SwOgVNgIRwyVU3pEEycBuQ", "LaEpxFGIQgCArsNZ2rd4Pw", > > "CiJ9gouZsbmTtxTWx7w6lA", "TaQV\_I01RfCq3B6uAtIBoQ", > > "9Jpjo5k-RlGfLVLF6nDgze", "57YpjRdASsrrae-RD3spog", > > "bmA4EWFSTiKUaDzaNcCFKQ", "Fui9z\_UbRe6AY1VhAr8Crw", > > "2PORr5BzSDOmBXgmQkO5Zg", "snfwTmtuTv-uj5mOWSJpgA", > > "0nHIrtePSaeW8aWArh\_Mrg", "s0g9QHnjTgWX3rCIu1g0Hg", > > "Jl67fACuQvCFgZxXAFtDOg" ], > > "\_cache" : true, > > "\_cache\_key" : "my\_terms\_cache" > > } > > } > > } > > > > I already used "_ids filter_" but I got same behaviour. One thing that I > > realized is that one of the cluster's nodes is increasing the Search Thread > > Pool (something like Queue: 50 and Count: 47) and the others don't > > (something like Queue: 0 and Count: 1). If I remove this node from the > > cluster another one starts with the same problem. > > > > My current environment is: > > > > - 7 Data nodes with 16Gb (8Gb for ES)and 8 cores each one; > > - 4 Load balancer Nodes (no data, no master) with 4Gb (3Gb for ES) and 8 > > cores each one; > > - 4 MasterNodes (only master, no data) with 4Gb (3Gb for ES) and 8 cores > > each one; > > - Thread Pool Search 47 (the others are standard config); > > - 7 Shards and 2 replicas Index; > > - 14.6Gb Index size (14.524.273 documents); > > > > I'm executing this filter with 50 concurrent users. > > > > Regards > > > > Em terça-feira, 3 de junho de 2014 20h33min45s UTC-3, Jörg Prante > > escreveu: > > > > > Can you show your test code? > > > > > > You seem to look at the wrong settings - by adjusting node number, shard > > > number, replica number alone, you can not find out the maximum node > > > performance. E.g. concurrency settings, index optimizations, query > > > optimizations, thread pooling, and most of all, fast disk subsystem I/O is > > > important. > > > > > > Jörg > > > > > > On Wed, Jun 4, 2014 at 12:18 AM, Marcelo Paes Rech \<[marcelo...@gmail.com](mailto:marcelo...@gmail.com) > > > > > > > wrote: > > > > > > > Thanks for your reply Nikolas. It helps a lot. > > > > > > > > And about the quantity of documents of each shard, or size of each > > > > shard. And the need of no data nodes or only master nodes. When is it > > > > necessary? > > > > > > > > Some tests I did, when I increased request's number (like 100 users at > > > > same moment, and redo it again and again), 5 nodes with 1 shard and 2 > > > > replicas each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The > > > > response time start to increase more than 5s (I think less than 1s, in > > > > this case, would be acceptable) . > > > > > > > > This test has a lot of documents (something like 14 millions). > > > > > > > > Thanks. Regards. > > > > > > > > Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas Everett > > > > escreveu: > > > > > > > > > On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech \< > > > > > [marcelo...@gmail.com](mailto:marcelo...@gmail.com)\> wrote: > > > > > > > > > > Hi guys, > > > > > > > > > > > I'm looking for an article or a guide for the best cluster > > > > > > configuration. I read a lot of articles like "change this configuration" > > > > > > and "you must create X shards per node" but I didn't saw nothing like > > > > > > Elasticsearch Official guide for creating a cluster. > > > > > > > > > > > > What I would like to know are informations like. > > > > > > > > > > > > - How to calculate how many shards will be good for the cluster. > > > > > > - How many shards do we need per node? And if this is variable, how > > > > > > do I calculate this? > > > > > > - How much memory do I need per node and how many nodes? > > > > > > > > > > > > I think Elasticsearch is well documentated. But it is very fragmented. > > > > > > > > > > For some of these that is because "it depends" is the answer. For > > > > > example, you'll want larger heaps for aggregations and faceting. > > > > > > > > > > There are some rules of thumb: > > > > > > > > > > 1. Set Elasticsearch's heap memory to 1/2 of ram but not more then > > > > > 30GB. Bigger then that and the JVM can't do pointer compression and you > > > > > effectively lose ram. > > > > > 2. #1 implies that having much more then 60GB of ram on each node > > > > > doesn't make a big difference. It helps but its not really as good as > > > > > having more nodes. > > > > > 3. The most efficient efficient way of sharding is likely one shard > > > > > on each node. So if you have 9 nodes and a replication factor of 2 (so 3 > > > > > total copies) then 3 shards is likely to be more efficient then having 2 or > > > > > 4. But this only really matters when those shards get lots of traffic. > > > > > And it breaks down a bit when you get lots of nodes. And the in presence > > > > > of routing. Its complicated. > > > > > > > > > > But these are really just starting points, safe-ish defaults. > > > > > > > > > > Nik > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [June 5, 2014, 12:46pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/8 "2014-06-05T12:46:11Z")
Ah, that is a simple resolution, thanks for highlighting it. Jörg On Thu, Jun 5, 2014 at 2:38 PM, Marcelo Paes Rech \<[marcelopaesrech@gmail.com](mailto:marcelopaesrech@gmail.com) > wrote: > Hi Jörg. Thanks for your reply again. > > As I said, I already had used ids filter, but I got the same behaviour. > > I realized what was wrong. Maybe it could be a bug in ES or not. When I > executed the filter I included "from" and "size" attibutes. In this case > "size" was 999999, but the final result would be just 10 documents. > Aparently ES pre-allocates the objects that I say I will use (maybe for > performance reasons), but if the final result is not the total (999999), ES > doesn't remove remaining pre-allocated objects until the memory (heap) is > full. > > I changed the size attribute to 10 and heap became stable. > > That's it. Thanks. > > Regards. > > Em quarta-feira, 4 de junho de 2014 19h54min15s UTC-3, Jörg Prante > escreveu: > > > Why do you use terms on \_id field and not the the ids filter? ids filter > > is more efficient since it reuses the \_uid field which is cached by default. > > > > Do the terms in the query vary from query to query? If so, caching might > > kill your heap. > > > > Another possible issue is that your query is not distributed to all > > shards, if the query does not vary from user to user in your test. If so, > > you created a "hot spot", all the load from the 100 users wold go to a > > limited number of node with a limited shard count. > > > > The search thread pool seems small with 50 searches if you execute > > searches for 100 users in parallel, this can lead to a congestion of the > > search module. Why don't you use 100 (at least)? > > > > Jörg > > > > On Wed, Jun 4, 2014 at 2:40 PM, Marcelo Paes Rech [marcelo...@gmail.com](mailto:marcelo...@gmail.com) > > wrote: > > > > > Hi Jörg. Thanks for your reply. > > > > > > Here is my filter. > > > {"filter": > > > { > > > "terms" : { > > > "\_id" : [ "QSxrbEM8TKe5zr8931xBjA", "wj63ghegRwC6qLsWq2chkA", > > > "hYEhDbAqQwSRxhYfvDgFkg", "4bZmPE1fTYqijphRyyWiuQ", > > > "Fhq53yYyT3CEw6vclKu\_NA", "XL2atBraTEyx57MefjFVhA", > > > "951i0dZkT064FlQkzHnnWA", "O8Ixbir1TrGT\_IA3wKfsHg", > > > "8k4U7KsuTmsThqxy-5YaKw", "GNOoQTHglf22kzcE7EOf8g", > > > "-RQeY48fTg2kYnh2M4E1cQ", "u8DGBdfVR9WRVj6d9E4Ebw", > > > "WFHSXd7UQvCMYFBhFcTsng", "qnQ7q7FyTsg397lM1EWgqA", > > > "wRQtUzdMRy2qOkMCNxdpgA", "Ll83iglxSUS\_Gs7mjkMt8w", > > > "d2sxZ1oBTfuvAfov5EJ0iw", "cyht-vB4Q-mMSg9N5jcGXg", > > > "bNSVaO47QTOCkfJhWo0qjg", "BHuhm55IRerKnynJ8WgFTw", > > > "fHKA4PF2QteWm8E7dW7CAw", "DLE6A7tyQJ-zcKcCa6IPSA", > > > "qfelTW7-SuGRQ0GKbngARA", "R7VHHJhYsUqfuxYof8BJ8w", > > > "W4PqiJfPSlSFjVKFsGkA4Q", "Juq62zOsRdheuW3O6Gb2KA", > > > "U9v0IKj\_RrgRNjE31ZTt2g", "uNHa0kOOT5qjPpzxZcs35A", > > > "SwOgVNgIRwyVU3pEEycBuQ", "LaEpxFGIQgCArsNZ2rd4Pw", > > > "CiJ9gouZsbmTtxTWx7w6lA", "TaQV\_I01RfCq3B6uAtIBoQ", > > > "9Jpjo5k-RlGfLVLF6nDgze", "57YpjRdASsrrae-RD3spog", > > > "bmA4EWFSTiKUaDzaNcCFKQ", "Fui9z\_UbRe6AY1VhAr8Crw", > > > "2PORr5BzSDOmBXgmQkO5Zg", "snfwTmtuTv-uj5mOWSJpgA", > > > "0nHIrtePSaeW8aWArh\_Mrg", "s0g9QHnjTgWX3rCIu1g0Hg", > > > "Jl67fACuQvCFgZxXAFtDOg" ], > > > "\_cache" : true, > > > "\_cache\_key" : "my\_terms\_cache" > > > } > > > } > > > } > > > > > > I already used "_ids filter_" but I got same behaviour. One thing that > > > I realized is that one of the cluster's nodes is increasing the Search > > > Thread Pool (something like Queue: 50 and Count: 47) and the others don't > > > (something like Queue: 0 and Count: 1). If I remove this node from the > > > cluster another one starts with the same problem. > > > > > > My current environment is: > > > > > > - 7 Data nodes with 16Gb (8Gb for ES)and 8 cores each one; > > > - 4 Load balancer Nodes (no data, no master) with 4Gb (3Gb for ES) and > > > 8 cores each one; > > > - 4 MasterNodes (only master, no data) with 4Gb (3Gb for ES) and 8 > > > cores each one; > > > - Thread Pool Search 47 (the others are standard config); > > > - 7 Shards and 2 replicas Index; > > > - 14.6Gb Index size (14.524.273 documents); > > > > > > I'm executing this filter with 50 concurrent users. > > > > > > Regards > > > > > > Em terça-feira, 3 de junho de 2014 20h33min45s UTC-3, Jörg Prante > > > escreveu: > > > > > > > Can you show your test code? > > > > > > > > You seem to look at the wrong settings - by adjusting node number, > > > > shard number, replica number alone, you can not find out the maximum node > > > > performance. E.g. concurrency settings, index optimizations, query > > > > optimizations, thread pooling, and most of all, fast disk subsystem I/O is > > > > important. > > > > > > > > Jörg > > > > > > > > On Wed, Jun 4, 2014 at 12:18 AM, Marcelo Paes Rech \< > > > > [marcelo...@gmail.com](mailto:marcelo...@gmail.com)\> wrote: > > > > > > > > > Thanks for your reply Nikolas. It helps a lot. > > > > > > > > > > And about the quantity of documents of each shard, or size of each > > > > > shard. And the need of no data nodes or only master nodes. When is it > > > > > necessary? > > > > > > > > > > Some tests I did, when I increased request's number (like 100 users at > > > > > same moment, and redo it again and again), 5 nodes with 1 shard and 2 > > > > > replicas each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The > > > > > response time start to increase more than 5s (I think less than 1s, in > > > > > this case, would be acceptable) . > > > > > > > > > > This test has a lot of documents (something like 14 millions). > > > > > > > > > > Thanks. Regards. > > > > > > > > > > Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas > > > > > Everett escreveu: > > > > > > > > > > > On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech \< > > > > > > [marcelo...@gmail.com](mailto:marcelo...@gmail.com)\> wrote: > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > I'm looking for an article or a guide for the best cluster > > > > > > > configuration. I read a lot of articles like "change this configuration" > > > > > > > and "you must create X shards per node" but I didn't saw nothing like > > > > > > > Elasticsearch Official guide for creating a cluster. > > > > > > > > > > > > > > What I would like to know are informations like. > > > > > > > > > > > > > > - How to calculate how many shards will be good for the cluster. > > > > > > > - How many shards do we need per node? And if this is variable, how > > > > > > > do I calculate this? > > > > > > > - How much memory do I need per node and how many nodes? > > > > > > > > > > > > > > I think Elasticsearch is well documentated. But it is very > > > > > > > fragmented. > > > > > > > > > > > > For some of these that is because "it depends" is the answer. For > > > > > > example, you'll want larger heaps for aggregations and faceting. > > > > > > > > > > > > There are some rules of thumb: > > > > > > > > > > > > 1. Set Elasticsearch's heap memory to 1/2 of ram but not more then > > > > > > 30GB. Bigger then that and the JVM can't do pointer compression and you > > > > > > effectively lose ram. > > > > > > 2. #1 implies that having much more then 60GB of ram on each node > > > > > > doesn't make a big difference. It helps but its not really as good as > > > > > > having more nodes. > > > > > > 3. The most efficient efficient way of sharding is likely one shard > > > > > > on each node. So if you have 9 nodes and a replication factor of 2 (so 3 > > > > > > total copies) then 3 shards is likely to be more efficient then having 2 or > > > > > > 4. But this only really matters when those shards get lots of traffic. > > > > > > And it breaks down a bit when you get lots of nodes. And the in presence > > > > > > of routing. Its complicated. > > > > > > > > > > > > But these are really just starting points, safe-ish defaults. > > > > > > > > > > > > Nik > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40goo > > > > > [glegroups.com](http://glegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEW6q%2BR%2BTVYGL\_21oDimEWiZUgUb\_gcLJUMBq6BLZkHnw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEW6q%2BR%2BTVYGL_21oDimEWiZUgUb_gcLJUMBq6BLZkHnw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 5, 2014, 11:01pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/9 "2014-06-05T23:01:33Z")
This would probably be worth raising as a github issue - > **[elasticsearch](https://github.com/elasticsearch/)** > > GitHub is where elasticsearch builds software. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 5 June 2014 22:38, Marcelo Paes Rech [marcelopaesrech@gmail.com](mailto:marcelopaesrech@gmail.com) wrote: > Hi Jörg. Thanks for your reply again. > > As I said, I already had used ids filter, but I got the same behaviour. > > I realized what was wrong. Maybe it could be a bug in ES or not. When I > executed the filter I included "from" and "size" attibutes. In this case > "size" was 999999, but the final result would be just 10 documents. > Aparently ES pre-allocates the objects that I say I will use (maybe for > performance reasons), but if the final result is not the total (999999), ES > doesn't remove remaining pre-allocated objects until the memory (heap) is > full. > > I changed the size attribute to 10 and heap became stable. > > That's it. Thanks. > > Regards. > > Em quarta-feira, 4 de junho de 2014 19h54min15s UTC-3, Jörg Prante > escreveu: > > > Why do you use terms on \_id field and not the the ids filter? ids filter > > is more efficient since it reuses the \_uid field which is cached by default. > > > > Do the terms in the query vary from query to query? If so, caching might > > kill your heap. > > > > Another possible issue is that your query is not distributed to all > > shards, if the query does not vary from user to user in your test. If so, > > you created a "hot spot", all the load from the 100 users wold go to a > > limited number of node with a limited shard count. > > > > The search thread pool seems small with 50 searches if you execute > > searches for 100 users in parallel, this can lead to a congestion of the > > search module. Why don't you use 100 (at least)? > > > > Jörg > > > > On Wed, Jun 4, 2014 at 2:40 PM, Marcelo Paes Rech [marcelo...@gmail.com](mailto:marcelo...@gmail.com) > > wrote: > > > > > Hi Jörg. Thanks for your reply. > > > > > > Here is my filter. > > > {"filter": > > > { > > > "terms" : { > > > "\_id" : [ "QSxrbEM8TKe5zr8931xBjA", "wj63ghegRwC6qLsWq2chkA", > > > "hYEhDbAqQwSRxhYfvDgFkg", "4bZmPE1fTYqijphRyyWiuQ", > > > "Fhq53yYyT3CEw6vclKu\_NA", "XL2atBraTEyx57MefjFVhA", > > > "951i0dZkT064FlQkzHnnWA", "O8Ixbir1TrGT\_IA3wKfsHg", > > > "8k4U7KsuTmsThqxy-5YaKw", "GNOoQTHglf22kzcE7EOf8g", > > > "-RQeY48fTg2kYnh2M4E1cQ", "u8DGBdfVR9WRVj6d9E4Ebw", > > > "WFHSXd7UQvCMYFBhFcTsng", "qnQ7q7FyTsg397lM1EWgqA", > > > "wRQtUzdMRy2qOkMCNxdpgA", "Ll83iglxSUS\_Gs7mjkMt8w", > > > "d2sxZ1oBTfuvAfov5EJ0iw", "cyht-vB4Q-mMSg9N5jcGXg", > > > "bNSVaO47QTOCkfJhWo0qjg", "BHuhm55IRerKnynJ8WgFTw", > > > "fHKA4PF2QteWm8E7dW7CAw", "DLE6A7tyQJ-zcKcCa6IPSA", > > > "qfelTW7-SuGRQ0GKbngARA", "R7VHHJhYsUqfuxYof8BJ8w", > > > "W4PqiJfPSlSFjVKFsGkA4Q", "Juq62zOsRdheuW3O6Gb2KA", > > > "U9v0IKj\_RrgRNjE31ZTt2g", "uNHa0kOOT5qjPpzxZcs35A", > > > "SwOgVNgIRwyVU3pEEycBuQ", "LaEpxFGIQgCArsNZ2rd4Pw", > > > "CiJ9gouZsbmTtxTWx7w6lA", "TaQV\_I01RfCq3B6uAtIBoQ", > > > "9Jpjo5k-RlGfLVLF6nDgze", "57YpjRdASsrrae-RD3spog", > > > "bmA4EWFSTiKUaDzaNcCFKQ", "Fui9z\_UbRe6AY1VhAr8Crw", > > > "2PORr5BzSDOmBXgmQkO5Zg", "snfwTmtuTv-uj5mOWSJpgA", > > > "0nHIrtePSaeW8aWArh\_Mrg", "s0g9QHnjTgWX3rCIu1g0Hg", > > > "Jl67fACuQvCFgZxXAFtDOg" ], > > > "\_cache" : true, > > > "\_cache\_key" : "my\_terms\_cache" > > > } > > > } > > > } > > > > > > I already used "_ids filter_" but I got same behaviour. One thing that > > > I realized is that one of the cluster's nodes is increasing the Search > > > Thread Pool (something like Queue: 50 and Count: 47) and the others don't > > > (something like Queue: 0 and Count: 1). If I remove this node from the > > > cluster another one starts with the same problem. > > > > > > My current environment is: > > > > > > - 7 Data nodes with 16Gb (8Gb for ES)and 8 cores each one; > > > - 4 Load balancer Nodes (no data, no master) with 4Gb (3Gb for ES) and > > > 8 cores each one; > > > - 4 MasterNodes (only master, no data) with 4Gb (3Gb for ES) and 8 > > > cores each one; > > > - Thread Pool Search 47 (the others are standard config); > > > - 7 Shards and 2 replicas Index; > > > - 14.6Gb Index size (14.524.273 documents); > > > > > > I'm executing this filter with 50 concurrent users. > > > > > > Regards > > > > > > Em terça-feira, 3 de junho de 2014 20h33min45s UTC-3, Jörg Prante > > > escreveu: > > > > > > > Can you show your test code? > > > > > > > > You seem to look at the wrong settings - by adjusting node number, > > > > shard number, replica number alone, you can not find out the maximum node > > > > performance. E.g. concurrency settings, index optimizations, query > > > > optimizations, thread pooling, and most of all, fast disk subsystem I/O is > > > > important. > > > > > > > > Jörg > > > > > > > > On Wed, Jun 4, 2014 at 12:18 AM, Marcelo Paes Rech \< > > > > [marcelo...@gmail.com](mailto:marcelo...@gmail.com)\> wrote: > > > > > > > > > Thanks for your reply Nikolas. It helps a lot. > > > > > > > > > > And about the quantity of documents of each shard, or size of each > > > > > shard. And the need of no data nodes or only master nodes. When is it > > > > > necessary? > > > > > > > > > > Some tests I did, when I increased request's number (like 100 users at > > > > > same moment, and redo it again and again), 5 nodes with 1 shard and 2 > > > > > replicas each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The > > > > > response time start to increase more than 5s (I think less than 1s, in > > > > > this case, would be acceptable) . > > > > > > > > > > This test has a lot of documents (something like 14 millions). > > > > > > > > > > Thanks. Regards. > > > > > > > > > > Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas > > > > > Everett escreveu: > > > > > > > > > > > On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech \< > > > > > > [marcelo...@gmail.com](mailto:marcelo...@gmail.com)\> wrote: > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > I'm looking for an article or a guide for the best cluster > > > > > > > configuration. I read a lot of articles like "change this configuration" > > > > > > > and "you must create X shards per node" but I didn't saw nothing like > > > > > > > Elasticsearch Official guide for creating a cluster. > > > > > > > > > > > > > > What I would like to know are informations like. > > > > > > > > > > > > > > - How to calculate how many shards will be good for the cluster. > > > > > > > - How many shards do we need per node? And if this is variable, how > > > > > > > do I calculate this? > > > > > > > - How much memory do I need per node and how many nodes? > > > > > > > > > > > > > > I think Elasticsearch is well documentated. But it is very > > > > > > > fragmented. > > > > > > > > > > > > For some of these that is because "it depends" is the answer. For > > > > > > example, you'll want larger heaps for aggregations and faceting. > > > > > > > > > > > > There are some rules of thumb: > > > > > > > > > > > > 1. Set Elasticsearch's heap memory to 1/2 of ram but not more then > > > > > > 30GB. Bigger then that and the JVM can't do pointer compression and you > > > > > > effectively lose ram. > > > > > > 2. #1 implies that having much more then 60GB of ram on each node > > > > > > doesn't make a big difference. It helps but its not really as good as > > > > > > having more nodes. > > > > > > 3. The most efficient efficient way of sharding is likely one shard > > > > > > on each node. So if you have 9 nodes and a replication factor of 2 (so 3 > > > > > > total copies) then 3 shards is likely to be more efficient then having 2 or > > > > > > 4. But this only really matters when those shards get lots of traffic. > > > > > > And it breaks down a bit when you get lots of nodes. And the in presence > > > > > > of routing. Its complicated. > > > > > > > > > > > > But these are really just starting points, safe-ish defaults. > > > > > > > > > > > > Nik > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40goo > > > > > [glegroups.com](http://glegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bcbP3uGf7\_n9QV3QpumDJ-vVrQ4x\_JPSV0-2TT80N4rg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bcbP3uGf7_n9QV3QpumDJ-vVrQ4x_JPSV0-2TT80N4rg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Marcelo\_Paes\_Rech](https://avatars.discourse-cdn.com/v4/letter/m/ba8739/32.png) [@Marcelo\_Paes\_Rech](https://discuss.elastic.co/u/Marcelo_Paes_Rech)\ **Post date:** [June 11, 2014, 12:49pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/10 "2014-06-11T12:49:50Z")
I have just created a issue: > > > When I do a filter or query, and set the size valeu to a huge value (like 100000…00), the memory consuption is increased a lot even though the final result be 10 documents. > > If I do the same thing with a huge amount of users the memory is increased so fast that the nodes would become bottlenecks (sometimes a node shutdowns itself). > > I solved that changing the size attribute to a low value, but I think it would be better if the ES deal with that in a better way, avoiding bad programmers to shutdown the server. > > Regards. Regards. Em quinta-feira, 5 de junho de 2014 20h02min05s UTC-3, Mark Walkom escreveu: > This would probably be worth raising as a github issue - > [elasticsearch · GitHub](https://github.com/elasticsearch/) > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 5 June 2014 22:38, Marcelo Paes Rech \<[marcelo...@gmail.com](mailto:marcelo...@gmail.com) > \\> wrote: > > > Hi Jörg. Thanks for your reply again. > > > > As I said, I already had used ids filter, but I got the same behaviour. > > > > I realized what was wrong. Maybe it could be a bug in ES or not. When I > > executed the filter I included "from" and "size" attibutes. In this case > > "size" was 999999, but the final result would be just 10 documents. > > Aparently ES pre-allocates the objects that I say I will use (maybe for > > performance reasons), but if the final result is not the total (999999), ES > > doesn't remove remaining pre-allocated objects until the memory (heap) is > > full. > > > > I changed the size attribute to 10 and heap became stable. > > > > That's it. Thanks. > > > > Regards. > > > > Em quarta-feira, 4 de junho de 2014 19h54min15s UTC-3, Jörg Prante > > escreveu: > > > > > Why do you use terms on \_id field and not the the ids filter? ids filter > > > is more efficient since it reuses the \_uid field which is cached by default. > > > > > > Do the terms in the query vary from query to query? If so, caching might > > > kill your heap. > > > > > > Another possible issue is that your query is not distributed to all > > > shards, if the query does not vary from user to user in your test. If so, > > > you created a "hot spot", all the load from the 100 users wold go to a > > > limited number of node with a limited shard count. > > > > > > The search thread pool seems small with 50 searches if you execute > > > searches for 100 users in parallel, this can lead to a congestion of the > > > search module. Why don't you use 100 (at least)? > > > > > > Jörg > > > > > > On Wed, Jun 4, 2014 at 2:40 PM, Marcelo Paes Rech [marcelo...@gmail.com](mailto:marcelo...@gmail.com) > > > wrote: > > > > > > > Hi Jörg. Thanks for your reply. > > > > > > > > Here is my filter. > > > > {"filter": > > > > { > > > > "terms" : { > > > > "\_id" : [ "QSxrbEM8TKe5zr8931xBjA", "wj63ghegRwC6qLsWq2chkA", > > > > "hYEhDbAqQwSRxhYfvDgFkg", "4bZmPE1fTYqijphRyyWiuQ", > > > > "Fhq53yYyT3CEw6vclKu\_NA", "XL2atBraTEyx57MefjFVhA", > > > > "951i0dZkT064FlQkzHnnWA", "O8Ixbir1TrGT\_IA3wKfsHg", > > > > "8k4U7KsuTmsThqxy-5YaKw", "GNOoQTHglf22kzcE7EOf8g", > > > > "-RQeY48fTg2kYnh2M4E1cQ", "u8DGBdfVR9WRVj6d9E4Ebw", > > > > "WFHSXd7UQvCMYFBhFcTsng", "qnQ7q7FyTsg397lM1EWgqA", > > > > "wRQtUzdMRy2qOkMCNxdpgA", "Ll83iglxSUS\_Gs7mjkMt8w", > > > > "d2sxZ1oBTfuvAfov5EJ0iw", "cyht-vB4Q-mMSg9N5jcGXg", > > > > "bNSVaO47QTOCkfJhWo0qjg", "BHuhm55IRerKnynJ8WgFTw", > > > > "fHKA4PF2QteWm8E7dW7CAw", "DLE6A7tyQJ-zcKcCa6IPSA", > > > > "qfelTW7-SuGRQ0GKbngARA", "R7VHHJhYsUqfuxYof8BJ8w", > > > > "W4PqiJfPSlSFjVKFsGkA4Q", "Juq62zOsRdheuW3O6Gb2KA", > > > > "U9v0IKj\_RrgRNjE31ZTt2g", "uNHa0kOOT5qjPpzxZcs35A", > > > > "SwOgVNgIRwyVU3pEEycBuQ", "LaEpxFGIQgCArsNZ2rd4Pw", > > > > "CiJ9gouZsbmTtxTWx7w6lA", "TaQV\_I01RfCq3B6uAtIBoQ", > > > > "9Jpjo5k-RlGfLVLF6nDgze", "57YpjRdASsrrae-RD3spog", > > > > "bmA4EWFSTiKUaDzaNcCFKQ", "Fui9z\_UbRe6AY1VhAr8Crw", > > > > "2PORr5BzSDOmBXgmQkO5Zg", "snfwTmtuTv-uj5mOWSJpgA", > > > > "0nHIrtePSaeW8aWArh\_Mrg", "s0g9QHnjTgWX3rCIu1g0Hg", > > > > "Jl67fACuQvCFgZxXAFtDOg" ], > > > > "\_cache" : true, > > > > "\_cache\_key" : "my\_terms\_cache" > > > > } > > > > } > > > > } > > > > > > > > I already used "_ids filter_" but I got same behaviour. One thing that > > > > I realized is that one of the cluster's nodes is increasing the Search > > > > Thread Pool (something like Queue: 50 and Count: 47) and the others don't > > > > (something like Queue: 0 and Count: 1). If I remove this node from the > > > > cluster another one starts with the same problem. > > > > > > > > My current environment is: > > > > > > > > - 7 Data nodes with 16Gb (8Gb for ES)and 8 cores each one; > > > > - 4 Load balancer Nodes (no data, no master) with 4Gb (3Gb for ES) and > > > > 8 cores each one; > > > > - 4 MasterNodes (only master, no data) with 4Gb (3Gb for ES) and 8 > > > > cores each one; > > > > - Thread Pool Search 47 (the others are standard config); > > > > - 7 Shards and 2 replicas Index; > > > > - 14.6Gb Index size (14.524.273 documents); > > > > > > > > I'm executing this filter with 50 concurrent users. > > > > > > > > Regards > > > > > > > > Em terça-feira, 3 de junho de 2014 20h33min45s UTC-3, Jörg Prante > > > > escreveu: > > > > > > > > > Can you show your test code? > > > > > > > > > > You seem to look at the wrong settings - by adjusting node number, > > > > > shard number, replica number alone, you can not find out the maximum node > > > > > performance. E.g. concurrency settings, index optimizations, query > > > > > optimizations, thread pooling, and most of all, fast disk subsystem I/O is > > > > > important. > > > > > > > > > > Jörg > > > > > > > > > > On Wed, Jun 4, 2014 at 12:18 AM, Marcelo Paes Rech \< > > > > > [marcelo...@gmail.com](mailto:marcelo...@gmail.com)\> wrote: > > > > > > > > > > > Thanks for your reply Nikolas. It helps a lot. > > > > > > > > > > > > And about the quantity of documents of each shard, or size of each > > > > > > shard. And the need of no data nodes or only master nodes. When is it > > > > > > necessary? > > > > > > > > > > > > Some tests I did, when I increased request's number (like 100 users > > > > > > at same moment, and redo it again and again), 5 nodes with 1 shard and 2 > > > > > > replicas each and 16Gb RAM (8Gb for ES and 8Gb for OS) weren't enough. The > > > > > > response time start to increase more than 5s (I think less than 1s, in > > > > > > this case, would be acceptable) . > > > > > > > > > > > > This test has a lot of documents (something like 14 millions). > > > > > > > > > > > > Thanks. Regards. > > > > > > > > > > > > Em segunda-feira, 2 de junho de 2014 17h09min04s UTC-3, Nikolas > > > > > > Everett escreveu: > > > > > > > > > > > > > On Mon, Jun 2, 2014 at 3:52 PM, Marcelo Paes Rech \< > > > > > > > [marcelo...@gmail.com](mailto:marcelo...@gmail.com)\> wrote: > > > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > > > I'm looking for an article or a guide for the best cluster > > > > > > > > configuration. I read a lot of articles like "change this configuration" > > > > > > > > and "you must create X shards per node" but I didn't saw nothing like > > > > > > > > Elasticsearch Official guide for creating a cluster. > > > > > > > > > > > > > > > > What I would like to know are informations like. > > > > > > > > > > > > > > > > - How to calculate how many shards will be good for the cluster. > > > > > > > > - How many shards do we need per node? And if this is variable, how > > > > > > > > do I calculate this? > > > > > > > > - How much memory do I need per node and how many nodes? > > > > > > > > > > > > > > > > I think Elasticsearch is well documentated. But it is very > > > > > > > > fragmented. > > > > > > > > > > > > > > For some of these that is because "it depends" is the answer. For > > > > > > > example, you'll want larger heaps for aggregations and faceting. > > > > > > > > > > > > > > There are some rules of thumb: > > > > > > > > > > > > > > 1. Set Elasticsearch's heap memory to 1/2 of ram but not more then > > > > > > > 30GB. Bigger then that and the JVM can't do pointer compression and you > > > > > > > effectively lose ram. > > > > > > > 2. #1 implies that having much more then 60GB of ram on each node > > > > > > > doesn't make a big difference. It helps but its not really as good as > > > > > > > having more nodes. > > > > > > > 3. The most efficient efficient way of sharding is likely one shard > > > > > > > on each node. So if you have 9 nodes and a replication factor of 2 (so 3 > > > > > > > total copies) then 3 shards is likely to be more efficient then having 2 or > > > > > > > 4. But this only really matters when those shards get lots of traffic. > > > > > > > And it breaks down a bit when you get lots of nodes. And the in presence > > > > > > > of routing. Its complicated. > > > > > > > > > > > > > > But these are really just starting points, safe-ish defaults. > > > > > > > > > > > > > > Nik > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40goo > > > > > > [glegroups.com](http://glegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/94b8ecf9-efc4-4046-a862-63b670ccc23e%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d32487cb-db92-4b7a-b6b3-afd431beaf61%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6f2f0e11-dd3d-4fc5-8adb-6eccddf83640%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c727d421-6426-4c52-b3ce-f8533e3dc68b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c727d421-6426-4c52-b3ce-f8533e3dc68b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 11, 2014, 3:21pm UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/11 "2014-06-11T15:21:55Z")
On Thursday, 5 June 2014 00:54:15 UTC+2, Jörg Prante wrote: > Why do you use terms on \_id field and not the the ids filter? ids filter > is more efficient since it reuses the \_uid field which is cached by default. So does the terms filter. The only advantage of the \_ids filter is that you can specify a type for each id, otherwise it looks for `type1#id`, `type2#id` etc, but that's the same for the ids filter if no type is specified. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6e80bac7-4957-40cc-935a-e9c027fb5941%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6e80bac7-4957-40cc-935a-e9c027fb5941%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:23am UTC](https://discuss.elastic.co/t/best-cluster-environment-for-search/17866/12 "2017-07-06T01:23:14Z")