# Need advice about configuring a cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [May 11, 2011, 2:15pm UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381 "2011-05-11T14:15:54Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Alois\_Cochard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alois_cochard/32/3176_2.png) [@Alois\_Cochard](https://discuss.elastic.co/u/Alois_Cochard)\ **Post date:** [May 11, 2011, 2:15pm UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/1 "2011-05-11T14:15:54Z")
Hello all, (would like to have your advice on this kimchy, I wonder what are your recommendation for big big indices on HPC infrastructure) I'm actually setting an ES cluster for QA, and moving from my tiny developer machine to the HPC infrastructure. I have at disposal 2 machines (or more for the future) with each: - 32 cores (8x4) - 32GB RAM - File System: GPFS [http://en.wikipedia.org/wiki/IBM\_General\_Parallel\_File\_System](http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System) For the moment I have 20million document in the first index and something like 100'000 in the second. The first index ('resource') could become really, really huge in the future (billions...). First topology I've setup: - 8 ES Nodes on each server (default RAM) - 16 shards per index and 1 replicate - Gateway: FS Why I choosed to run multiple ES node on a single server: - Preventing too long GC - Parallelize write on the FS (think at GPFS as having each folder on a separate disk) After discussing with lukasvlcek, I've come to this second solution: - 1 Node on each server - Gateway: Shared FS - Store: Memory (outside of the heap to prevent GC) - Huge number of shards per index (to be able to add nodes in the future) - No replica (no need if shared FS?) So my questions are: - Does it make sense 😉 any other ideas ? - Can ES do heavy parallel writing on the FS (without running multiple node on same machine) ? - How many max memory can ES handle (with the outside heap store) ? - How many shards did you recommend ? Thanks a lot guys for your help ! Alois ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [May 11, 2011, 6:05pm UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/2 "2011-05-11T18:05:12Z")
We have been running in production since November with ~31M docs and I will share our experiences. We initially started with a gateway file system and switched to local file system. First, the network utilization can be extremely high during index builds, even to the point on 1G ethernet to saturate the network causing nodes to become disconnected from the cluster. Yes, we could have switched to 10G or separate FS and data networks, or ganged connections. The network issues along with a single point of failure, convinced us to switch. Today we have 4 machines with 1 replica, which we found to be too low. We're increasing this to every machine having a full index (knowing this limits us in total index size and potential performance, but makes us most tolerant to network partitions). ES will make effective use of memory. We set ES to 50% of available physical ram (48G), to provide sufficient space OS cache and our applications (indexing and searching). We have several smaller indexes, rather than 1 large index, so our shard counts are moderate. We also expect moderate growth for any single index. Early one we would see a GC which would spike search results every ~20 hours. Since enabling the following setting in elasticseach.yml, we have not seen GC issues. # Force all memory to be prealloced and locked, forcing JVM to never swap bootstrap: mlockall: true The sharding should be thought of in the context of how many machines would I like my cluster to scale to? And, of the machines in the eventual cluster size, how many should be involved in a single query? This will be tied to how large the index is (# docs X doc size). David ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [May 11, 2011, 6:10pm UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/3 "2011-05-11T18:10:52Z")
Dne středa, 11. května 2011, Alois Cochard [alois.cochard@gmail.com](mailto:alois.cochard@gmail.com) napsal(a): > Hello all, > > (would like to have your advice on this kimchy, I wonder what are your > recommendation for big big indices on HPC infrastructure) > > I'm actually setting an ES cluster for QA, and moving from my tiny > developer machine to the HPC infrastructure. > > I have at disposal 2 machines (or more for the future) with each: > > - 32 cores (8x4) > - 32GB RAM > - File System: GPFS [GPFS - Wikipedia](http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System) > > For the moment I have 20million document in the first index and > something like 100'000 in the second. > The first index ('resource') could become really, really huge in the > future (billions...). > > First topology I've setup: > > - 8 ES Nodes on each server (default RAM) > - 16 shards per index and 1 replicate > - Gateway: FS > > Why I choosed to run multiple ES node on a single server: > > - Preventing too long GC > - Parallelize write on the FS (think at GPFS as having each folder on > a separate disk) > > After discussing with lukasvlcek, I've come to this second solution: > > - 1 Node on each server > - Gateway: Shared FS > - Store: Memory (outside of the heap to prevent GC) > - Huge number of shards per index (to be able to add nodes in the > future) > - No replica (no need if shared FS?) Depending on what huge means here, shards are good for scalability. If ES can then it allocates shards on different nodes so if you do not plan to have more then say 20 nodes in the future then starting with 100 shards or more per index probably does not make a lot of sense. Replicas are good for HA so if you start with two nodes then having one replica can be useful I think, if one machine crashes then you still have all data, just the health status would be yellow (which means that ES can not allocate all replicas but that is still fine for both searching and indexing). If I remember correctly the question was if having more shards of the same index on a single node results in faster indexing (because there are separed independent Lucene indices on that node). I am not really sure (Shay?) but there is also the transaction log file used in ES (if I understand it correctly then changes are acumulated into this file first and then flushed into a real Lucene index) so this can allow high parallel modififaction of a single index, thus having just one shard of given index per node is ok (but as I said this is just my speculation). > So my questions are: > > - Does it make sense 😉 any other ideas ? > - Can ES do heavy parallel writing on the FS (without running multiple > node on same machine) ? > - How many max memory can ES handle (with the outside heap store) ? > - How many shards did you recommend ? > > Thanks a lot guys for your help ! > > Alois ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 12, 2011, 9:13am UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/4 "2011-05-12T09:13:18Z")
Answers below: On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote: > Hello all, > > (would like to have your advice on this kimchy, I wonder what are your > recommendation for big big indices on HPC infrastructure) > > I'm actually setting an ES cluster for QA, and moving from my tiny > developer machine to the HPC infrastructure. > > I have at disposal 2 machines (or more for the future) with each: > > - 32 cores (8x4) > - 32GB RAM > - File System: GPFS [GPFS - Wikipedia](http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System) > I would say go with fast local disks, simpler and cheaper. > > For the moment I have 20million document in the first index and > something like 100'000 in the second. > The first index ('resource') could become really, really huge in the > future (billions...). > > First topology I've setup: > > - 8 ES Nodes on each server (default RAM) > 8 nodes won't buy you much. I would go with 1 node and ~16gb memory allocated to ES (min and max). > - 16 shards per index and 1 replicate > For the index that will get really large, that can be a good number (16 shards). You need to do some capacity planning to see how much a shard "costs" in your config. > - Gateway: FS I would recommend local gateway. As mentioned in another answer to this mail, the shared FS does cost in more bandwidth used. > Why I choosed to run multiple ES node on a single server: > > - Preventing too long GC > ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, but I don't think you will need to. > - Parallelize write on the FS (think at GPFS as having each folder on > a separate disk) > You can have the local data location stored on GPFS, if its really faster then local (possibly RAIDed) disk. > > After discussing with lukasvlcek, I've come to this second solution: > > - 1 Node on each server > - Gateway: Shared FS > See my point above regarding shared FS. I would say go with local gateway. > - Store: Memory (outside of the heap to prevent GC) > I would still use the file system storage, you don't have enough mem to store teh index fully in memory and it won't buy you that much with file system caching. > - Huge number of shards per index (to be able to add nodes in the > future) > You will need to do some capacity planning. A 16 shard index means that the index can grow up to 16 machines size, which is considerable. Another option is to check if the large index can be further partitioned to several indices that you can add on demand. > - No replica (no need if shared FS?) > Replicas are needed even with shared gateway for fast failover. > > So my questions are: > > - Does it make sense 😉 any other ideas ? > - Can ES do heavy parallel writing on the FS (without running multiple > node on same machine) ? > Yes, a single node is parallel writing to the FS. More shards on a given node can give it a boost if its a beefy machine. > - How many max memory can ES handle (with the outside heap store) ? > Indices are stored outside the heap, and thats up to the memory the machine has. But, I recommend using the FS to store the index. If you have enough mem, you can always mmap it. > - How many shards did you recommend ? > For the large index, 16 sounds like a good number, but without capacity planning, its hard to tell. For the smaller index, you can have much less. > > Thanks a lot guys for your help ! > > Alois ---
**Author:** ![Alois\_Cochard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alois_cochard/32/3176_2.png) [@Alois\_Cochard](https://discuss.elastic.co/u/Alois_Cochard)\ **Post date:** [May 16, 2011, 8:54am UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/5 "2011-05-16T08:54:54Z")
Hello, Thanks to all for your answers and your help ! I'm actually going for the local FS solution but did some test with memory store (+ shared FS) too. Deployment is in standby due to the need of raising the 'file opens' limits, I'll keep you in touch with the result of my tests. Cheers, On May 12, 11:13 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote: > > > Hello all, > > > (would like to have your advice on this kimchy, I wonder what are your > > recommendation for big big indices on HPC infrastructure) > > > I'm actually setting an ES cluster for QA, and moving from my tiny > > developer machine to the HPC infrastructure. > > > I have at disposal 2 machines (or more for the future) with each: > > > > - 32 cores (8x4) > > - 32GB RAM > > - File System: GPFShttp://en.wikipedia.org/wiki/IBM\_General\_Parallel\_File\_System > > I would say go with fast local disks, simpler and cheaper. > > > For the moment I have 20million document in the first index and > > something like 100'000 in the second. > > The first index ('resource') could become really, really huge in the > > future (billions...). > > > First topology I've setup: > > > > - 8 ES Nodes on each server (default RAM) > > 8 nodes won't buy you much. I would go with 1 node and ~16gb memory allocated to ES (min and max).\> - 16 shards per index and 1 replicate > > For the index that will get really large, that can be a good number (16 shards). You need to do some capacity planning to see how much a shard "costs" in your config. > > > - Gateway: FS > > I would recommend local gateway. As mentioned in another answer to this mail, the shared FS does cost in more bandwidth used. > > > Why I choosed to run multiple ES node on a single server: > > > > - Preventing too long GC > > ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, but I don't think you will need to.\> - Parallelize write on the FS (think at GPFS as having each folder on > > > a separate disk) > > You can have the local data location stored on GPFS, if its really faster then local (possibly RAIDed) disk. > > > After discussing with lukasvlcek, I've come to this second solution: > > > > - 1 Node on each server > > - Gateway: Shared FS > > See my point above regarding shared FS. I would say go with local gateway.\> - Store: Memory (outside of the heap to prevent GC) > > I would still use the file system storage, you don't have enough mem to store teh index fully in memory and it won't buy you that much with file system caching.\> - Huge number of shards per index (to be able to add nodes in the > > > future) > > You will need to do some capacity planning. A 16 shard index means that the index can grow up to 16 machines size, which is considerable. Another option is to check if the large index can be further partitioned to several indices that you can add on demand.\> - No replica (no need if shared FS?) > > Replicas are needed even with shared gateway for fast failover. > > > So my questions are: > > > > - Does it make sense 😉 any other ideas ? > > - Can ES do heavy parallel writing on the FS (without running multiple > > node on same machine) ? > > Yes, a single node is parallel writing to the FS. More shards on a given node can give it a boost if its a beefy machine.\> - How many max memory can ES handle (with the outside heap store) ? > > Indices are stored outside the heap, and thats up to the memory the machine has. But, I recommend using the FS to store the index. If you have enough mem, you can always mmap it.\> - How many shards did you recommend ? > > For the large index, 16 sounds like a good number, but without capacity planning, its hard to tell. For the smaller index, you can have much less. > > > Thanks a lot guys for your help ! > > > Alois ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [May 16, 2011, 9:06am UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/6 "2011-05-16T09:06:45Z")
Hey, I would like to extend this discussion by the following question: If the physical machines that are running ES nodes do not have any local file system - in fact they have a only mounted file system over NFS or by other means - does it make sense to use local gateway as well or it is better to go directly to shared gateway? As far as I understand the traffic will go over network in both cases, so is there any advantage in using local gateway in such case? Regards, Lukas On Mon, May 16, 2011 at 10:54 AM, Alois Cochard [alois.cochard@gmail.com](mailto:alois.cochard@gmail.com)wrote: > Hello, > > Thanks to all for your answers and your help ! > > I'm actually going for the local FS solution but did some test with > memory store (+ shared FS) too. > > Deployment is in standby due to the need of raising the 'file opens' > limits, I'll keep you in touch with the result of my tests. > > Cheers, > > On May 12, 11:13 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote: > > > > > Hello all, > > > > > (would like to have your advice on this kimchy, I wonder what are your > > > recommendation for big big indices on HPC infrastructure) > > > > > I'm actually setting an ES cluster for QA, and moving from my tiny > > > developer machine to the HPC infrastructure. > > > > > I have at disposal 2 machines (or more for the future) with each: > > > > > > - 32 cores (8x4) > > > - 32GB RAM > > > - File System: GPFShttp:// > > > [GPFS - Wikipedia](http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System) > > > > I would say go with fast local disks, simpler and cheaper. > > > > > For the moment I have 20million document in the first index and > > > something like 100'000 in the second. > > > The first index ('resource') could become really, really huge in the > > > future (billions...). > > > > > First topology I've setup: > > > > > > - 8 ES Nodes on each server (default RAM) > > > > 8 nodes won't buy you much. I would go with 1 node and ~16gb memory > > allocated to ES (min and max).\> - 16 shards per index and 1 replicate > > > > For the index that will get really large, that can be a good number (16 > > shards). You need to do some capacity planning to see how much a shard > > "costs" in your config. > > > > > - Gateway: FS > > > > I would recommend local gateway. As mentioned in another answer to this > > mail, the shared FS does cost in more bandwidth used. > > > > > Why I choosed to run multiple ES node on a single server: > > > > > > - Preventing too long GC > > > > ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, > > but I don't think you will need to.\> - Parallelize write on the FS (think at > > GPFS as having each folder on > > > > > a separate disk) > > > > You can have the local data location stored on GPFS, if its really faster > > then local (possibly RAIDed) disk. > > > > > After discussing with lukasvlcek, I've come to this second solution: > > > > > > - 1 Node on each server > > > - Gateway: Shared FS > > > > See my point above regarding shared FS. I would say go with local > > gateway.\> - Store: Memory (outside of the heap to prevent GC) > > > > I would still use the file system storage, you don't have enough mem to > > store teh index fully in memory and it won't buy you that much with file > > system caching.\> - Huge number of shards per index (to be able to add nodes > > in the > > > > > future) > > > > You will need to do some capacity planning. A 16 shard index means that > > the index can grow up to 16 machines size, which is considerable. Another > > option is to check if the large index can be further partitioned to several > > indices that you can add on demand.\> - No replica (no need if shared FS?) > > > > Replicas are needed even with shared gateway for fast failover. > > > > > So my questions are: > > > > > > - Does it make sense 😉 any other ideas ? > > > - Can ES do heavy parallel writing on the FS (without running multiple > > > node on same machine) ? > > > > Yes, a single node is parallel writing to the FS. More shards on a given > > node can give it a boost if its a beefy machine.\> - How many max memory can > > ES handle (with the outside heap store) ? > > > > Indices are stored outside the heap, and thats up to the memory the > > machine has. But, I recommend using the FS to store the index. If you have > > enough mem, you can always mmap it.\> - How many shards did you recommend ? > > > > For the large index, 16 sounds like a good number, but without capacity > > planning, its hard to tell. For the smaller index, you can have much less. > > > > > Thanks a lot guys for your help ! > > > > > Alois ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 16, 2011, 9:31am UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/7 "2011-05-16T09:31:22Z")
It will still write it twice in case of shared gateway and data dir on NFS. Once to the data dir, and then snapshot it to the gateway location. On Monday, May 16, 2011 at 12:06 PM, Lukáš Vlček wrote: > Hey, > > I would like to extend this discussion by the following question: > > If the physical machines that are running ES nodes do not have any local file system - in fact they have a only mounted file system over NFS or by other means - does it make sense to use local gateway as well or it is better to go directly to shared gateway? As far as I understand the traffic will go over network in both cases, so is there any advantage in using local gateway in such case? > > Regards, > Lukas > > On Mon, May 16, 2011 at 10:54 AM, Alois Cochard [alois.cochard@gmail.com](mailto:alois.cochard@gmail.com) wrote: > > > Hello, > > > > Thanks to all for your answers and your help ! > > > > I'm actually going for the local FS solution but did some test with > > memory store (+ shared FS) too. > > > > Deployment is in standby due to the need of raising the 'file opens' > > limits, I'll keep you in touch with the result of my tests. > > > > Cheers, > > > > On May 12, 11:13 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote: > > > > > > > Hello all, > > > > > > > (would like to have your advice on this kimchy, I wonder what are your > > > > recommendation for big big indices on HPC infrastructure) > > > > > > > I'm actually setting an ES cluster for QA, and moving from my tiny > > > > developer machine to the HPC infrastructure. > > > > > > > I have at disposal 2 machines (or more for the future) with each: > > > > > > > > - 32 cores (8x4) > > > > - 32GB RAM > > > > - File System: GPFShttp://en.wikipedia.org/wiki/IBM\_General\_Parallel\_File\_System > > > > > > I would say go with fast local disks, simpler and cheaper. > > > > > > > For the moment I have 20million document in the first index and > > > > something like 100'000 in the second. > > > > The first index ('resource') could become really, really huge in the > > > > future (billions...). > > > > > > > First topology I've setup: > > > > > > > > - 8 ES Nodes on each server (default RAM) > > > > > > 8 nodes won't buy you much. I would go with 1 node and ~16gb memory allocated to ES (min and max).\> - 16 shards per index and 1 replicate > > > > > > For the index that will get really large, that can be a good number (16 shards). You need to do some capacity planning to see how much a shard "costs" in your config. > > > > > > > - Gateway: FS > > > > > > I would recommend local gateway. As mentioned in another answer to this mail, the shared FS does cost in more bandwidth used. > > > > > > > Why I choosed to run multiple ES node on a single server: > > > > > > > > - Preventing too long GC > > > > > > ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, but I don't think you will need to.\> - Parallelize write on the FS (think at GPFS as having each folder on > > > > > > > a separate disk) > > > > > > You can have the local data location stored on GPFS, if its really faster then local (possibly RAIDed) disk. > > > > > > > After discussing with lukasvlcek, I've come to this second solution: > > > > > > > > - 1 Node on each server > > > > - Gateway: Shared FS > > > > > > See my point above regarding shared FS. I would say go with local gateway.\> - Store: Memory (outside of the heap to prevent GC) > > > > > > I would still use the file system storage, you don't have enough mem to store teh index fully in memory and it won't buy you that much with file system caching.\> - Huge number of shards per index (to be able to add nodes in the > > > > > > > future) > > > > > > You will need to do some capacity planning. A 16 shard index means that the index can grow up to 16 machines size, which is considerable. Another option is to check if the large index can be further partitioned to several indices that you can add on demand.\> - No replica (no need if shared FS?) > > > > > > Replicas are needed even with shared gateway for fast failover. > > > > > > > So my questions are: > > > > > > > > - Does it make sense 😉 any other ideas ? > > > > - Can ES do heavy parallel writing on the FS (without running multiple > > > > node on same machine) ? > > > > > > Yes, a single node is parallel writing to the FS. More shards on a given node can give it a boost if its a beefy machine.\> - How many max memory can ES handle (with the outside heap store) ? > > > > > > Indices are stored outside the heap, and thats up to the memory the machine has. But, I recommend using the FS to store the index. If you have enough mem, you can always mmap it.\> - How many shards did you recommend ? > > > > > > For the large index, 16 sounds like a good number, but without capacity planning, its hard to tell. For the smaller index, you can have much less. > > > > > > > Thanks a lot guys for your help ! > > > > > > > Alois ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [May 16, 2011, 10:13am UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/8 "2011-05-16T10:13:42Z")
But there is probably no way around this... if the machine has only network disks then the data will be always transmitted twice over the network, right? No matter if the gateway is of local or shared type. So my question was if there is any recommendation which gateway type is better in such case. Can local gateway perform a little better due to the fact that there will be no locking? On Mon, May 16, 2011 at 11:31 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > It will still write it twice in case of shared gateway and data dir on > NFS. Once to the data dir, and then snapshot it to the gateway location. > > On Monday, May 16, 2011 at 12:06 PM, Lukáš Vlček wrote: > > Hey, > > I would like to extend this discussion by the following question: > > If the physical machines that are running ES nodes do not have any local > file system - in fact they have a only mounted file system over NFS or by > other means - does it make sense to use local gateway as well or it is > better to go directly to shared gateway? As far as I understand the traffic > will go over network in both cases, so is there any advantage in using local > gateway in such case? > > Regards, > Lukas > > On Mon, May 16, 2011 at 10:54 AM, Alois Cochard [alois.cochard@gmail.com](mailto:alois.cochard@gmail.com)wrote: > > Hello, > > Thanks to all for your answers and your help ! > > I'm actually going for the local FS solution but did some test with > memory store (+ shared FS) too. > > Deployment is in standby due to the need of raising the 'file opens' > limits, I'll keep you in touch with the result of my tests. > > Cheers, > > On May 12, 11:13 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote: > > > > > Hello all, > > > > > (would like to have your advice on this kimchy, I wonder what are your > > > recommendation for big big indices on HPC infrastructure) > > > > > I'm actually setting an ES cluster for QA, and moving from my tiny > > > developer machine to the HPC infrastructure. > > > > > I have at disposal 2 machines (or more for the future) with each: > > > > > > - 32 cores (8x4) > > > - 32GB RAM > > > - File System: GPFShttp:// > > > [GPFS - Wikipedia](http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System) > > > > I would say go with fast local disks, simpler and cheaper. > > > > > For the moment I have 20million document in the first index and > > > something like 100'000 in the second. > > > The first index ('resource') could become really, really huge in the > > > future (billions...). > > > > > First topology I've setup: > > > > > > - 8 ES Nodes on each server (default RAM) > > > > 8 nodes won't buy you much. I would go with 1 node and ~16gb memory > > allocated to ES (min and max).\> - 16 shards per index and 1 replicate > > > > For the index that will get really large, that can be a good number (16 > > shards). You need to do some capacity planning to see how much a shard > > "costs" in your config. > > > > > - Gateway: FS > > > > I would recommend local gateway. As mentioned in another answer to this > > mail, the shared FS does cost in more bandwidth used. > > > > > Why I choosed to run multiple ES node on a single server: > > > > > > - Preventing too long GC > > > > ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, > > but I don't think you will need to.\> - Parallelize write on the FS (think at > > GPFS as having each folder on > > > > > a separate disk) > > > > You can have the local data location stored on GPFS, if its really faster > > then local (possibly RAIDed) disk. > > > > > After discussing with lukasvlcek, I've come to this second solution: > > > > > > - 1 Node on each server > > > - Gateway: Shared FS > > > > See my point above regarding shared FS. I would say go with local > > gateway.\> - Store: Memory (outside of the heap to prevent GC) > > > > I would still use the file system storage, you don't have enough mem to > > store teh index fully in memory and it won't buy you that much with file > > system caching.\> - Huge number of shards per index (to be able to add nodes > > in the > > > > > future) > > > > You will need to do some capacity planning. A 16 shard index means that > > the index can grow up to 16 machines size, which is considerable. Another > > option is to check if the large index can be further partitioned to several > > indices that you can add on demand.\> - No replica (no need if shared FS?) > > > > Replicas are needed even with shared gateway for fast failover. > > > > > So my questions are: > > > > > > - Does it make sense 😉 any other ideas ? > > > - Can ES do heavy parallel writing on the FS (without running multiple > > > node on same machine) ? > > > > Yes, a single node is parallel writing to the FS. More shards on a given > > node can give it a boost if its a beefy machine.\> - How many max memory can > > ES handle (with the outside heap store) ? > > > > Indices are stored outside the heap, and thats up to the memory the > > machine has. But, I recommend using the FS to store the index. If you have > > enough mem, you can always mmap it.\> - How many shards did you recommend ? > > > > For the large index, 16 sounds like a good number, but without capacity > > planning, its hard to tell. For the smaller index, you can have much less. > > > > > Thanks a lot guys for your help ! > > > > > Alois ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 16, 2011, 10:15am UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/9 "2011-05-16T10:15:44Z")
No, if you use local gateway, and the data location is on a network drive, then it will be "transmitted" (during indexing) only once. On Monday, May 16, 2011 at 1:13 PM, Lukáš Vlček wrote: > But there is probably no way around this... if the machine has only network disks then the data will be always transmitted twice over the network, right? No matter if the gateway is of local or shared type. So my question was if there is any recommendation which gateway type is better in such case. Can local gateway perform a little better due to the fact that there will be no locking? > > On Mon, May 16, 2011 at 11:31 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > It will still write it twice in case of shared gateway and data dir on NFS. Once to the data dir, and then snapshot it to the gateway location. > > On Monday, May 16, 2011 at 12:06 PM, Lukáš Vlček wrote: > > > > > Hey, > > > > > > I would like to extend this discussion by the following question: > > > > > > If the physical machines that are running ES nodes do not have any local file system - in fact they have a only mounted file system over NFS or by other means - does it make sense to use local gateway as well or it is better to go directly to shared gateway? As far as I understand the traffic will go over network in both cases, so is there any advantage in using local gateway in such case? > > > > > > Regards, > > > Lukas > > > > > > On Mon, May 16, 2011 at 10:54 AM, Alois Cochard [alois.cochard@gmail.com](mailto:alois.cochard@gmail.com) wrote: > > > > > > > Hello, > > > > > > > > Thanks to all for your answers and your help ! > > > > > > > > I'm actually going for the local FS solution but did some test with > > > > memory store (+ shared FS) too. > > > > > > > > Deployment is in standby due to the need of raising the 'file opens' > > > > limits, I'll keep you in touch with the result of my tests. > > > > > > > > Cheers, > > > > > > > > On May 12, 11:13 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > > > Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote: > > > > > > > > > > > Hello all, > > > > > > > > > > > (would like to have your advice on this kimchy, I wonder what are your > > > > > > recommendation for big big indices on HPC infrastructure) > > > > > > > > > > > I'm actually setting an ES cluster for QA, and moving from my tiny > > > > > > developer machine to the HPC infrastructure. > > > > > > > > > > > I have at disposal 2 machines (or more for the future) with each: > > > > > > > > > > > > - 32 cores (8x4) > > > > > > - 32GB RAM > > > > > > - File System: GPFShttp://en.wikipedia.org/wiki/IBM\_General\_Parallel\_File\_System > > > > > > > > > > I would say go with fast local disks, simpler and cheaper. > > > > > > > > > > > For the moment I have 20million document in the first index and > > > > > > something like 100'000 in the second. > > > > > > The first index ('resource') could become really, really huge in the > > > > > > future (billions...). > > > > > > > > > > > First topology I've setup: > > > > > > > > > > > > - 8 ES Nodes on each server (default RAM) > > > > > > > > > > 8 nodes won't buy you much. I would go with 1 node and ~16gb memory allocated to ES (min and max).\> - 16 shards per index and 1 replicate > > > > > > > > > > For the index that will get really large, that can be a good number (16 shards). You need to do some capacity planning to see how much a shard "costs" in your config. > > > > > > > > > > > - Gateway: FS > > > > > > > > > > I would recommend local gateway. As mentioned in another answer to this mail, the shared FS does cost in more bandwidth used. > > > > > > > > > > > Why I choosed to run multiple ES node on a single server: > > > > > > > > > > > > - Preventing too long GC > > > > > > > > > > ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, but I don't think you will need to.\> - Parallelize write on the FS (think at GPFS as having each folder on > > > > > > > > > > > a separate disk) > > > > > > > > > > You can have the local data location stored on GPFS, if its really faster then local (possibly RAIDed) disk. > > > > > > > > > > > After discussing with lukasvlcek, I've come to this second solution: > > > > > > > > > > > > - 1 Node on each server > > > > > > - Gateway: Shared FS > > > > > > > > > > See my point above regarding shared FS. I would say go with local gateway.\> - Store: Memory (outside of the heap to prevent GC) > > > > > > > > > > I would still use the file system storage, you don't have enough mem to store teh index fully in memory and it won't buy you that much with file system caching.\> - Huge number of shards per index (to be able to add nodes in the > > > > > > > > > > > future) > > > > > > > > > > You will need to do some capacity planning. A 16 shard index means that the index can grow up to 16 machines size, which is considerable. Another option is to check if the large index can be further partitioned to several indices that you can add on demand.\> - No replica (no need if shared FS?) > > > > > > > > > > Replicas are needed even with shared gateway for fast failover. > > > > > > > > > > > So my questions are: > > > > > > > > > > > > - Does it make sense 😉 any other ideas ? > > > > > > - Can ES do heavy parallel writing on the FS (without running multiple > > > > > > node on same machine) ? > > > > > > > > > > Yes, a single node is parallel writing to the FS. More shards on a given node can give it a boost if its a beefy machine.\> - How many max memory can ES handle (with the outside heap store) ? > > > > > > > > > > Indices are stored outside the heap, and thats up to the memory the machine has. But, I recommend using the FS to store the index. If you have enough mem, you can always mmap it.\> - How many shards did you recommend ? > > > > > > > > > > For the large index, 16 sounds like a good number, but without capacity planning, its hard to tell. For the smaller index, you can have much less. > > > > > > > > > > > Thanks a lot guys for your help ! > > > > > > > > > > > Alois ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:06am UTC](https://discuss.elastic.co/t/need-advice-about-configuring-a-cluster/4381/10 "2017-07-06T04:06:09Z")