ES allocate too much RAM when using mmapfs

Elastic Stack Elasticsearch
1

Hi,

We have a problem with ES servers on production - ES allocate almost all
the RAM and eventually ES is not responding.
We have 2 nodes, each one with Windows Server 2008, 65 GB of RAM, JVM heap
size set to 20 GB, Windows page file set to 20 GB.
The index size is about 200 GB with 5 shards and 1 replica.
The index.storage.type is set to mmapfs.

The problem is that after about one week, the memory used by ES increase
from 20 GB to 60!
How can we solve this problem? set the index.storage.type to simplefs? add
more RAM? or can we limit the amount of memory allocated to ES?

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

mmapfs allocates all available OS memory automatically for best read
performance to Lucene files and this is controlled by the OS. Note, even
if the memory is fully utilized, it is still available for the OS for
memory management and other tasks on demand, so there is no need to
worry. If you have 65 GB of RAM and all the RAM is used, this is exactly
how mmapfs should work.

But there might be bugs. For Windows Server 2008, you need a hotfix
"33" DOS error code when memory memory-mapped files are cleaned by using the FlushViewOfFile() function in Windows 7 or in Windows Server 2008 R2 - Microsoft Support (I'm Linux enthusiastic - this
hint is taken from MongoDB Windows installation page
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/
which memory demand is similar to ES)

If you want default Java heap-limited amount of index files in memory,
select niofs or simplefs.

Jörg

Am 16.07.13 11:40, schrieb Amit Bh:

Hi,

We have a problem with ES servers on production - ES allocate almost
all the RAM and eventually ES is not responding.
We have 2 nodes, each one with Windows Server 2008, 65 GB of RAM, JVM
heap size set to 20 GB, Windows page file set to 20 GB.
The index size is about 200 GB with 5 shards and 1 replica.
The index.storage.type is set to mmapfs.

The problem is that after about one week, the memory used by ES
increase from 20 GB to 60!
How can we solve this problem? set the index.storage.type to simplefs?
add more RAM? or can we limit the amount of memory allocated to ES?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks allot Jörg!
We installed the hotfix on both servers and I hope it will solved the
problem.
We also increased the virtual memory from 20 GB to 90 GB, do you think it
might be related and can affect the memory?

Thanks again :slight_smile:

On Tuesday, July 16, 2013 12:40:41 PM UTC+3, Amit Bh wrote:

Hi,

We have a problem with ES servers on production - ES allocate almost all
the RAM and eventually ES is not responding.
We have 2 nodes, each one with Windows Server 2008, 65 GB of RAM, JVM heap
size set to 20 GB, Windows page file set to 20 GB.
The index size is about 200 GB with 5 shards and 1 replica.
The index.storage.type is set to mmapfs.

The problem is that after about one week, the memory used by ES increase
from 20 GB to 60!
How can we solve this problem? set the index.storage.type to simplefs? add
more RAM? or can we limit the amount of memory allocated to ES?

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.