Need advice about configuring a cluster

Elastic Stack Elasticsearch
1

Hello all,

(would like to have your advice on this kimchy, I wonder what are your
recommendation for big big indices on HPC infrastructure)

I'm actually setting an ES cluster for QA, and moving from my tiny
developer machine to the HPC infrastructure.

I have at disposal 2 machines (or more for the future) with each:

For the moment I have 20million document in the first index and
something like 100'000 in the second.
The first index ('resource') could become really, really huge in the
future (billions...).

First topology I've setup:

  • 8 ES Nodes on each server (default RAM)
  • 16 shards per index and 1 replicate
  • Gateway: FS

Why I choosed to run multiple ES node on a single server:

  • Preventing too long GC
  • Parallelize write on the FS (think at GPFS as having each folder on
    a separate disk)

After discussing with lukasvlcek, I've come to this second solution:

  • 1 Node on each server
  • Gateway: Shared FS
  • Store: Memory (outside of the heap to prevent GC)
  • Huge number of shards per index (to be able to add nodes in the
    future)
  • No replica (no need if shared FS?)

So my questions are:

  • Does it make sense :wink: any other ideas ?
  • Can ES do heavy parallel writing on the FS (without running multiple
    node on same machine) ?
  • How many max memory can ES handle (with the outside heap store) ?
  • How many shards did you recommend ?

Thanks a lot guys for your help !

Alois

2

We have been running in production since November with ~31M docs and I
will share our experiences.

We initially started with a gateway file system and switched to local
file system. First, the network utilization can be extremely high
during index builds, even to the point on 1G ethernet to saturate the
network causing nodes to become disconnected from the cluster. Yes, we
could have switched to 10G or separate FS and data networks, or ganged
connections. The network issues along with a single point of failure,
convinced us to switch. Today we have 4 machines with 1 replica, which
we found to be too low. We're increasing this to every machine having
a full index (knowing this limits us in total index size and potential
performance, but makes us most tolerant to network partitions).

ES will make effective use of memory. We set ES to 50% of available
physical ram (48G), to provide sufficient space OS cache and our
applications (indexing and searching). We have several smaller
indexes, rather than 1 large index, so our shard counts are moderate.
We also expect moderate growth for any single index.

Early one we would see a GC which would spike search results every ~20
hours. Since enabling the following setting in elasticseach.yml, we
have not seen GC issues.

Force all memory to be prealloced and locked, forcing JVM to never

swap
bootstrap:
mlockall: true

The sharding should be thought of in the context of how many machines
would I like my cluster to scale to? And, of the machines in the
eventual cluster size, how many should be involved in a single query?
This will be tied to how large the index is (# docs X doc size).

David

3

Dne středa, 11. května 2011, Alois Cochard alois.cochard@gmail.com napsal(a):

Hello all,

(would like to have your advice on this kimchy, I wonder what are your
recommendation for big big indices on HPC infrastructure)

I'm actually setting an ES cluster for QA, and moving from my tiny
developer machine to the HPC infrastructure.

I have at disposal 2 machines (or more for the future) with each:

For the moment I have 20million document in the first index and
something like 100'000 in the second.
The first index ('resource') could become really, really huge in the
future (billions...).

First topology I've setup:

  • 8 ES Nodes on each server (default RAM)
  • 16 shards per index and 1 replicate
  • Gateway: FS

Why I choosed to run multiple ES node on a single server:

  • Preventing too long GC
  • Parallelize write on the FS (think at GPFS as having each folder on
    a separate disk)

After discussing with lukasvlcek, I've come to this second solution:

  • 1 Node on each server
  • Gateway: Shared FS
  • Store: Memory (outside of the heap to prevent GC)
  • Huge number of shards per index (to be able to add nodes in the
    future)
  • No replica (no need if shared FS?)

Depending on what huge means here, shards are good for scalability. If
ES can then it allocates shards on different nodes so if you do not
plan to have more then say 20 nodes in the future then starting with
100 shards or more per index probably does not make a lot of sense.
Replicas are good for HA so if you start with two nodes then having
one replica can be useful I think, if one machine crashes then you
still have all data, just the health status would be yellow (which
means that ES can not allocate all replicas but that is still fine for
both searching and indexing).

If I remember correctly the question was if having more shards of the
same index on a single node results in faster indexing (because there
are separed independent Lucene indices on that node). I am not really
sure (Shay?) but there is also the transaction log file used in ES (if
I understand it correctly then changes are acumulated into this file
first and then flushed into a real Lucene index) so this can allow
high parallel modififaction of a single index, thus having just one
shard of given index per node is ok (but as I said this is just my
speculation).

So my questions are:

  • Does it make sense :wink: any other ideas ?
  • Can ES do heavy parallel writing on the FS (without running multiple
    node on same machine) ?
  • How many max memory can ES handle (with the outside heap store) ?
  • How many shards did you recommend ?

Thanks a lot guys for your help !

Alois

4

Answers below:
On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote:

Hello all,

(would like to have your advice on this kimchy, I wonder what are your
recommendation for big big indices on HPC infrastructure)

I'm actually setting an ES cluster for QA, and moving from my tiny
developer machine to the HPC infrastructure.

I have at disposal 2 machines (or more for the future) with each:

  • 32 cores (8x4)
  • 32GB RAM
  • File System: GPFS GPFS - Wikipedia
    I would say go with fast local disks, simpler and cheaper.

For the moment I have 20million document in the first index and
something like 100'000 in the second.
The first index ('resource') could become really, really huge in the
future (billions...).

First topology I've setup:

  • 8 ES Nodes on each server (default RAM)
    8 nodes won't buy you much. I would go with 1 node and ~16gb memory allocated to ES (min and max).
  • 16 shards per index and 1 replicate
    For the index that will get really large, that can be a good number (16 shards). You need to do some capacity planning to see how much a shard "costs" in your config.
  • Gateway: FS

I would recommend local gateway. As mentioned in another answer to this mail, the shared FS does cost in more bandwidth used.

Why I choosed to run multiple ES node on a single server:

  • Preventing too long GC
    ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, but I don't think you will need to.
  • Parallelize write on the FS (think at GPFS as having each folder on
    a separate disk)
    You can have the local data location stored on GPFS, if its really faster then local (possibly RAIDed) disk.

After discussing with lukasvlcek, I've come to this second solution:

  • 1 Node on each server
  • Gateway: Shared FS
    See my point above regarding shared FS. I would say go with local gateway.
  • Store: Memory (outside of the heap to prevent GC)
    I would still use the file system storage, you don't have enough mem to store teh index fully in memory and it won't buy you that much with file system caching.
  • Huge number of shards per index (to be able to add nodes in the
    future)
    You will need to do some capacity planning. A 16 shard index means that the index can grow up to 16 machines size, which is considerable. Another option is to check if the large index can be further partitioned to several indices that you can add on demand.
  • No replica (no need if shared FS?)
    Replicas are needed even with shared gateway for fast failover.

So my questions are:

  • Does it make sense :wink: any other ideas ?
  • Can ES do heavy parallel writing on the FS (without running multiple
    node on same machine) ?
    Yes, a single node is parallel writing to the FS. More shards on a given node can give it a boost if its a beefy machine.
  • How many max memory can ES handle (with the outside heap store) ?
    Indices are stored outside the heap, and thats up to the memory the machine has. But, I recommend using the FS to store the index. If you have enough mem, you can always mmap it.
  • How many shards did you recommend ?
    For the large index, 16 sounds like a good number, but without capacity planning, its hard to tell. For the smaller index, you can have much less.

Thanks a lot guys for your help !

Alois

5

Hello,

Thanks to all for your answers and your help !

I'm actually going for the local FS solution but did some test with
memory store (+ shared FS) too.

Deployment is in standby due to the need of raising the 'file opens'
limits, I'll keep you in touch with the result of my tests.

Cheers,

On May 12, 11:13 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote:

Hello all,

(would like to have your advice on this kimchy, I wonder what are your
recommendation for big big indices on HPC infrastructure)

I'm actually setting an ES cluster for QA, and moving from my tiny
developer machine to the HPC infrastructure.

I have at disposal 2 machines (or more for the future) with each:

  • 32 cores (8x4)
  • 32GB RAM
  • File System: GPFShttp://en.wikipedia.org/wiki/IBM_General_Parallel_File_System

I would say go with fast local disks, simpler and cheaper.

For the moment I have 20million document in the first index and
something like 100'000 in the second.
The first index ('resource') could become really, really huge in the
future (billions...).

First topology I've setup:

  • 8 ES Nodes on each server (default RAM)

8 nodes won't buy you much. I would go with 1 node and ~16gb memory allocated to ES (min and max).> - 16 shards per index and 1 replicate

For the index that will get really large, that can be a good number (16 shards). You need to do some capacity planning to see how much a shard "costs" in your config.

  • Gateway: FS

I would recommend local gateway. As mentioned in another answer to this mail, the shared FS does cost in more bandwidth used.

Why I choosed to run multiple ES node on a single server:

  • Preventing too long GC

ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, but I don't think you will need to.> - Parallelize write on the FS (think at GPFS as having each folder on

a separate disk)

You can have the local data location stored on GPFS, if its really faster then local (possibly RAIDed) disk.

After discussing with lukasvlcek, I've come to this second solution:

  • 1 Node on each server
  • Gateway: Shared FS

See my point above regarding shared FS. I would say go with local gateway.> - Store: Memory (outside of the heap to prevent GC)

I would still use the file system storage, you don't have enough mem to store teh index fully in memory and it won't buy you that much with file system caching.> - Huge number of shards per index (to be able to add nodes in the

future)

You will need to do some capacity planning. A 16 shard index means that the index can grow up to 16 machines size, which is considerable. Another option is to check if the large index can be further partitioned to several indices that you can add on demand.> - No replica (no need if shared FS?)

Replicas are needed even with shared gateway for fast failover.

So my questions are:

  • Does it make sense :wink: any other ideas ?
  • Can ES do heavy parallel writing on the FS (without running multiple
    node on same machine) ?

Yes, a single node is parallel writing to the FS. More shards on a given node can give it a boost if its a beefy machine.> - How many max memory can ES handle (with the outside heap store) ?

Indices are stored outside the heap, and thats up to the memory the machine has. But, I recommend using the FS to store the index. If you have enough mem, you can always mmap it.> - How many shards did you recommend ?

For the large index, 16 sounds like a good number, but without capacity planning, its hard to tell. For the smaller index, you can have much less.

Thanks a lot guys for your help !

Alois

6

Hey,

I would like to extend this discussion by the following question:

If the physical machines that are running ES nodes do not have any local
file system - in fact they have a only mounted file system over NFS or by
other means - does it make sense to use local gateway as well or it is
better to go directly to shared gateway? As far as I understand the traffic
will go over network in both cases, so is there any advantage in using local
gateway in such case?

Regards,
Lukas

On Mon, May 16, 2011 at 10:54 AM, Alois Cochard alois.cochard@gmail.comwrote:

Hello,

Thanks to all for your answers and your help !

I'm actually going for the local FS solution but did some test with
memory store (+ shared FS) too.

Deployment is in standby due to the need of raising the 'file opens'
limits, I'll keep you in touch with the result of my tests.

Cheers,

On May 12, 11:13 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote:

Hello all,

(would like to have your advice on this kimchy, I wonder what are your
recommendation for big big indices on HPC infrastructure)

I'm actually setting an ES cluster for QA, and moving from my tiny
developer machine to the HPC infrastructure.

I have at disposal 2 machines (or more for the future) with each:

I would say go with fast local disks, simpler and cheaper.

For the moment I have 20million document in the first index and
something like 100'000 in the second.
The first index ('resource') could become really, really huge in the
future (billions...).

First topology I've setup:

  • 8 ES Nodes on each server (default RAM)

8 nodes won't buy you much. I would go with 1 node and ~16gb memory
allocated to ES (min and max).> - 16 shards per index and 1 replicate

For the index that will get really large, that can be a good number (16
shards). You need to do some capacity planning to see how much a shard
"costs" in your config.

  • Gateway: FS

I would recommend local gateway. As mentioned in another answer to this
mail, the shared FS does cost in more bandwidth used.

Why I choosed to run multiple ES node on a single server:

  • Preventing too long GC

ES strives to be a good GC citizen. You can run 2 nodes each with 8gb,
but I don't think you will need to.> - Parallelize write on the FS (think at
GPFS as having each folder on

a separate disk)

You can have the local data location stored on GPFS, if its really faster
then local (possibly RAIDed) disk.

After discussing with lukasvlcek, I've come to this second solution:

  • 1 Node on each server
  • Gateway: Shared FS

See my point above regarding shared FS. I would say go with local
gateway.> - Store: Memory (outside of the heap to prevent GC)

I would still use the file system storage, you don't have enough mem to
store teh index fully in memory and it won't buy you that much with file
system caching.> - Huge number of shards per index (to be able to add nodes
in the

future)

You will need to do some capacity planning. A 16 shard index means that
the index can grow up to 16 machines size, which is considerable. Another
option is to check if the large index can be further partitioned to several
indices that you can add on demand.> - No replica (no need if shared FS?)

Replicas are needed even with shared gateway for fast failover.

So my questions are:

  • Does it make sense :wink: any other ideas ?
  • Can ES do heavy parallel writing on the FS (without running multiple
    node on same machine) ?

Yes, a single node is parallel writing to the FS. More shards on a given
node can give it a boost if its a beefy machine.> - How many max memory can
ES handle (with the outside heap store) ?

Indices are stored outside the heap, and thats up to the memory the
machine has. But, I recommend using the FS to store the index. If you have
enough mem, you can always mmap it.> - How many shards did you recommend ?

For the large index, 16 sounds like a good number, but without capacity
planning, its hard to tell. For the smaller index, you can have much less.

Thanks a lot guys for your help !

Alois

7

It will still write it twice in case of shared gateway and data dir on NFS. Once to the data dir, and then snapshot it to the gateway location.
On Monday, May 16, 2011 at 12:06 PM, Lukáš Vlček wrote:

Hey,

I would like to extend this discussion by the following question:

If the physical machines that are running ES nodes do not have any local file system - in fact they have a only mounted file system over NFS or by other means - does it make sense to use local gateway as well or it is better to go directly to shared gateway? As far as I understand the traffic will go over network in both cases, so is there any advantage in using local gateway in such case?

Regards,
Lukas

On Mon, May 16, 2011 at 10:54 AM, Alois Cochard alois.cochard@gmail.com wrote:

Hello,

Thanks to all for your answers and your help !

I'm actually going for the local FS solution but did some test with
memory store (+ shared FS) too.

Deployment is in standby due to the need of raising the 'file opens'
limits, I'll keep you in touch with the result of my tests.

Cheers,

On May 12, 11:13 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote:

Hello all,

(would like to have your advice on this kimchy, I wonder what are your
recommendation for big big indices on HPC infrastructure)

I'm actually setting an ES cluster for QA, and moving from my tiny
developer machine to the HPC infrastructure.

I have at disposal 2 machines (or more for the future) with each:

  • 32 cores (8x4)
  • 32GB RAM
  • File System: GPFShttp://en.wikipedia.org/wiki/IBM_General_Parallel_File_System

I would say go with fast local disks, simpler and cheaper.

For the moment I have 20million document in the first index and
something like 100'000 in the second.
The first index ('resource') could become really, really huge in the
future (billions...).

First topology I've setup:

  • 8 ES Nodes on each server (default RAM)

8 nodes won't buy you much. I would go with 1 node and ~16gb memory allocated to ES (min and max).> - 16 shards per index and 1 replicate

For the index that will get really large, that can be a good number (16 shards). You need to do some capacity planning to see how much a shard "costs" in your config.

  • Gateway: FS

I would recommend local gateway. As mentioned in another answer to this mail, the shared FS does cost in more bandwidth used.

Why I choosed to run multiple ES node on a single server:

  • Preventing too long GC

ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, but I don't think you will need to.> - Parallelize write on the FS (think at GPFS as having each folder on

a separate disk)

You can have the local data location stored on GPFS, if its really faster then local (possibly RAIDed) disk.

After discussing with lukasvlcek, I've come to this second solution:

  • 1 Node on each server
  • Gateway: Shared FS

See my point above regarding shared FS. I would say go with local gateway.> - Store: Memory (outside of the heap to prevent GC)

I would still use the file system storage, you don't have enough mem to store teh index fully in memory and it won't buy you that much with file system caching.> - Huge number of shards per index (to be able to add nodes in the

future)

You will need to do some capacity planning. A 16 shard index means that the index can grow up to 16 machines size, which is considerable. Another option is to check if the large index can be further partitioned to several indices that you can add on demand.> - No replica (no need if shared FS?)

Replicas are needed even with shared gateway for fast failover.

So my questions are:

  • Does it make sense :wink: any other ideas ?
  • Can ES do heavy parallel writing on the FS (without running multiple
    node on same machine) ?

Yes, a single node is parallel writing to the FS. More shards on a given node can give it a boost if its a beefy machine.> - How many max memory can ES handle (with the outside heap store) ?

Indices are stored outside the heap, and thats up to the memory the machine has. But, I recommend using the FS to store the index. If you have enough mem, you can always mmap it.> - How many shards did you recommend ?

For the large index, 16 sounds like a good number, but without capacity planning, its hard to tell. For the smaller index, you can have much less.

Thanks a lot guys for your help !

Alois

8

But there is probably no way around this... if the machine has only network
disks then the data will be always transmitted twice over the network,
right? No matter if the gateway is of local or shared type. So my question
was if there is any recommendation which gateway type is better in such
case. Can local gateway perform a little better due to the fact that there
will be no locking?

On Mon, May 16, 2011 at 11:31 AM, Shay Banon
shay.banon@elasticsearch.comwrote:

It will still write it twice in case of shared gateway and data dir on
NFS. Once to the data dir, and then snapshot it to the gateway location.

On Monday, May 16, 2011 at 12:06 PM, Lukáš Vlček wrote:

Hey,

I would like to extend this discussion by the following question:

If the physical machines that are running ES nodes do not have any local
file system - in fact they have a only mounted file system over NFS or by
other means - does it make sense to use local gateway as well or it is
better to go directly to shared gateway? As far as I understand the traffic
will go over network in both cases, so is there any advantage in using local
gateway in such case?

Regards,
Lukas

On Mon, May 16, 2011 at 10:54 AM, Alois Cochard alois.cochard@gmail.comwrote:

Hello,

Thanks to all for your answers and your help !

I'm actually going for the local FS solution but did some test with
memory store (+ shared FS) too.

Deployment is in standby due to the need of raising the 'file opens'
limits, I'll keep you in touch with the result of my tests.

Cheers,

On May 12, 11:13 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote:

Hello all,

(would like to have your advice on this kimchy, I wonder what are your
recommendation for big big indices on HPC infrastructure)

I'm actually setting an ES cluster for QA, and moving from my tiny
developer machine to the HPC infrastructure.

I have at disposal 2 machines (or more for the future) with each:

I would say go with fast local disks, simpler and cheaper.

For the moment I have 20million document in the first index and
something like 100'000 in the second.
The first index ('resource') could become really, really huge in the
future (billions...).

First topology I've setup:

  • 8 ES Nodes on each server (default RAM)

8 nodes won't buy you much. I would go with 1 node and ~16gb memory
allocated to ES (min and max).> - 16 shards per index and 1 replicate

For the index that will get really large, that can be a good number (16
shards). You need to do some capacity planning to see how much a shard
"costs" in your config.

  • Gateway: FS

I would recommend local gateway. As mentioned in another answer to this
mail, the shared FS does cost in more bandwidth used.

Why I choosed to run multiple ES node on a single server:

  • Preventing too long GC

ES strives to be a good GC citizen. You can run 2 nodes each with 8gb,
but I don't think you will need to.> - Parallelize write on the FS (think at
GPFS as having each folder on

a separate disk)

You can have the local data location stored on GPFS, if its really faster
then local (possibly RAIDed) disk.

After discussing with lukasvlcek, I've come to this second solution:

  • 1 Node on each server
  • Gateway: Shared FS

See my point above regarding shared FS. I would say go with local
gateway.> - Store: Memory (outside of the heap to prevent GC)

I would still use the file system storage, you don't have enough mem to
store teh index fully in memory and it won't buy you that much with file
system caching.> - Huge number of shards per index (to be able to add nodes
in the

future)

You will need to do some capacity planning. A 16 shard index means that
the index can grow up to 16 machines size, which is considerable. Another
option is to check if the large index can be further partitioned to several
indices that you can add on demand.> - No replica (no need if shared FS?)

Replicas are needed even with shared gateway for fast failover.

So my questions are:

  • Does it make sense :wink: any other ideas ?
  • Can ES do heavy parallel writing on the FS (without running multiple
    node on same machine) ?

Yes, a single node is parallel writing to the FS. More shards on a given
node can give it a boost if its a beefy machine.> - How many max memory can
ES handle (with the outside heap store) ?

Indices are stored outside the heap, and thats up to the memory the
machine has. But, I recommend using the FS to store the index. If you have
enough mem, you can always mmap it.> - How many shards did you recommend ?

For the large index, 16 sounds like a good number, but without capacity
planning, its hard to tell. For the smaller index, you can have much less.

Thanks a lot guys for your help !

Alois

9

No, if you use local gateway, and the data location is on a network drive, then it will be "transmitted" (during indexing) only once.
On Monday, May 16, 2011 at 1:13 PM, Lukáš Vlček wrote:

But there is probably no way around this... if the machine has only network disks then the data will be always transmitted twice over the network, right? No matter if the gateway is of local or shared type. So my question was if there is any recommendation which gateway type is better in such case. Can local gateway perform a little better due to the fact that there will be no locking?

On Mon, May 16, 2011 at 11:31 AM, Shay Banon shay.banon@elasticsearch.com wrote:

It will still write it twice in case of shared gateway and data dir on NFS. Once to the data dir, and then snapshot it to the gateway location.
On Monday, May 16, 2011 at 12:06 PM, Lukáš Vlček wrote:

Hey,

I would like to extend this discussion by the following question:

If the physical machines that are running ES nodes do not have any local file system - in fact they have a only mounted file system over NFS or by other means - does it make sense to use local gateway as well or it is better to go directly to shared gateway? As far as I understand the traffic will go over network in both cases, so is there any advantage in using local gateway in such case?

Regards,
Lukas

On Mon, May 16, 2011 at 10:54 AM, Alois Cochard alois.cochard@gmail.com wrote:

Hello,

Thanks to all for your answers and your help !

I'm actually going for the local FS solution but did some test with
memory store (+ shared FS) too.

Deployment is in standby due to the need of raising the 'file opens'
limits, I'll keep you in touch with the result of my tests.

Cheers,

On May 12, 11:13 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Answers below:On Wednesday, May 11, 2011 at 5:15 PM, Alois Cochard wrote:

Hello all,

(would like to have your advice on this kimchy, I wonder what are your
recommendation for big big indices on HPC infrastructure)

I'm actually setting an ES cluster for QA, and moving from my tiny
developer machine to the HPC infrastructure.

I have at disposal 2 machines (or more for the future) with each:

  • 32 cores (8x4)
  • 32GB RAM
  • File System: GPFShttp://en.wikipedia.org/wiki/IBM_General_Parallel_File_System

I would say go with fast local disks, simpler and cheaper.

For the moment I have 20million document in the first index and
something like 100'000 in the second.
The first index ('resource') could become really, really huge in the
future (billions...).

First topology I've setup:

  • 8 ES Nodes on each server (default RAM)

8 nodes won't buy you much. I would go with 1 node and ~16gb memory allocated to ES (min and max).> - 16 shards per index and 1 replicate

For the index that will get really large, that can be a good number (16 shards). You need to do some capacity planning to see how much a shard "costs" in your config.

  • Gateway: FS

I would recommend local gateway. As mentioned in another answer to this mail, the shared FS does cost in more bandwidth used.

Why I choosed to run multiple ES node on a single server:

  • Preventing too long GC

ES strives to be a good GC citizen. You can run 2 nodes each with 8gb, but I don't think you will need to.> - Parallelize write on the FS (think at GPFS as having each folder on

a separate disk)

You can have the local data location stored on GPFS, if its really faster then local (possibly RAIDed) disk.

After discussing with lukasvlcek, I've come to this second solution:

  • 1 Node on each server
  • Gateway: Shared FS

See my point above regarding shared FS. I would say go with local gateway.> - Store: Memory (outside of the heap to prevent GC)

I would still use the file system storage, you don't have enough mem to store teh index fully in memory and it won't buy you that much with file system caching.> - Huge number of shards per index (to be able to add nodes in the

future)

You will need to do some capacity planning. A 16 shard index means that the index can grow up to 16 machines size, which is considerable. Another option is to check if the large index can be further partitioned to several indices that you can add on demand.> - No replica (no need if shared FS?)

Replicas are needed even with shared gateway for fast failover.

So my questions are:

  • Does it make sense :wink: any other ideas ?
  • Can ES do heavy parallel writing on the FS (without running multiple
    node on same machine) ?

Yes, a single node is parallel writing to the FS. More shards on a given node can give it a boost if its a beefy machine.> - How many max memory can ES handle (with the outside heap store) ?

Indices are stored outside the heap, and thats up to the memory the machine has. But, I recommend using the FS to store the index. If you have enough mem, you can always mmap it.> - How many shards did you recommend ?

For the large index, 16 sounds like a good number, but without capacity planning, its hard to tell. For the smaller index, you can have much less.

Thanks a lot guys for your help !

Alois