What configuration is available to control MemoryMapDirectory

Hey guys,

We're running into some problems under heavy write, nominal read volume
when the Lucene memory mapped files have exhausted available physical
memory, and segments from disk must be paged into memory.

Are there any configs available to control how much physical memory is
available to MemoryMapDirectory?

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

There may not be much you can do here beyond giving the system more RAM and
moving to SSD.
How bad is the problem?

On 13 March 2015 at 17:03, Lindsey Poole lpoole@gmail.com wrote:

Hey guys,

We're running into some problems under heavy write, nominal read volume
when the Lucene memory mapped files have exhausted available physical
memory, and segments from disk must be paged into memory.

Are there any configs available to control how much physical memory is
available to MemoryMapDirectory?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8zSG55xY6SUF%2B9iB%3DTo4m%2Bv8VxgX2iUe8%3DaoqSPmBxGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

You may try limit direct memory on JVM level by
using -XX:MaxDirectMemorySize (default is unlimited). See also
ES_DIRECT_SIZE in
http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-service.html#_linux

I recommend at least 2GB

Jörg

On Sat, Mar 14, 2015 at 1:03 AM, Lindsey Poole lpoole@gmail.com wrote:

Hey guys,

We're running into some problems under heavy write, nominal read volume
when the Lucene memory mapped files have exhausted available physical
memory, and segments from disk must be paged into memory.

Are there any configs available to control how much physical memory is
available to MemoryMapDirectory?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH1q7WdOhPWo9D8vdCDVU97epZhATOU6qZyt_sVS_ySeg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I did see the ES_DIRECT_SIZE, but it seems to be ineffective.

I will try setting -XX:MaxDirectMemorySize directly.

On Saturday, March 14, 2015 at 4:43:22 AM UTC-7, Jörg Prante wrote:

You may try limit direct memory on JVM level by
using -XX:MaxDirectMemorySize (default is unlimited). See also
ES_DIRECT_SIZE in
http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-service.html#_linux

I recommend at least 2GB

Jörg

On Sat, Mar 14, 2015 at 1:03 AM, Lindsey Poole <lpo...@gmail.com
<javascript:>> wrote:

Hey guys,

We're running into some problems under heavy write, nominal read volume
when the Lucene memory mapped files have exhausted available physical
memory, and segments from disk must be paged into memory.

Are there any configs available to control how much physical memory is
available to MemoryMapDirectory?

Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0f540631-52f0-48d3-b33a-855c54d8ce94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

btw - we're on EC2 I2-4xl hosts, so we have ~120g ram and SSDs.

On Saturday, March 14, 2015 at 9:04:34 AM UTC-7, Lindsey Poole wrote:

I did see the ES_DIRECT_SIZE, but it seems to be ineffective.

I will try setting -XX:MaxDirectMemorySize directly.

On Saturday, March 14, 2015 at 4:43:22 AM UTC-7, Jörg Prante wrote:

You may try limit direct memory on JVM level by
using -XX:MaxDirectMemorySize (default is unlimited). See also
ES_DIRECT_SIZE in
http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-service.html#_linux

I recommend at least 2GB

Jörg

On Sat, Mar 14, 2015 at 1:03 AM, Lindsey Poole lpo...@gmail.com wrote:

Hey guys,

We're running into some problems under heavy write, nominal read volume
when the Lucene memory mapped files have exhausted available physical
memory, and segments from disk must be paged into memory.

Are there any configs available to control how much physical memory is
available to MemoryMapDirectory?

Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cf50ed46-cacf-414b-8b20-b82595dc2fd0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I'm out - no experience with EC2. I avoid foreign servers at all cost.
Maybe 120G RAM is affected by swap/memory overcommit. Do not forget to
check memlock and memory ballooning. The chances are few you can control
host settings as a guest in a virtual server environment.

Jörg

On Sat, Mar 14, 2015 at 5:06 PM, Lindsey Poole lpoole@gmail.com wrote:

btw - we're on EC2 I2-4xl hosts, so we have ~120g ram and SSDs.

On Saturday, March 14, 2015 at 9:04:34 AM UTC-7, Lindsey Poole wrote:

I did see the ES_DIRECT_SIZE, but it seems to be ineffective.

I will try setting -XX:MaxDirectMemorySize directly.

On Saturday, March 14, 2015 at 4:43:22 AM UTC-7, Jörg Prante wrote:

You may try limit direct memory on JVM level by
using -XX:MaxDirectMemorySize (default is unlimited). See also
ES_DIRECT_SIZE in http://www.elastic.co/guide/en/elasticsearch/
reference/current/setup-service.html#_linux

I recommend at least 2GB

Jörg

On Sat, Mar 14, 2015 at 1:03 AM, Lindsey Poole lpo...@gmail.com wrote:

Hey guys,

We're running into some problems under heavy write, nominal read volume
when the Lucene memory mapped files have exhausted available physical
memory, and segments from disk must be paged into memory.

Are there any configs available to control how much physical memory is
available to MemoryMapDirectory?

Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cf50ed46-cacf-414b-8b20-b82595dc2fd0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cf50ed46-cacf-414b-8b20-b82595dc2fd0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHY_M2BAywrG%2BaNcg59xA4_ocph1oqE0bzba4HTqrdLqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Can you provide more info on what the error/problem is, logs might help.

On 14 March 2015 at 10:12, joergprante@gmail.com joergprante@gmail.com
wrote:

I'm out - no experience with EC2. I avoid foreign servers at all cost.
Maybe 120G RAM is affected by swap/memory overcommit. Do not forget to
check memlock and memory ballooning. The chances are few you can control
host settings as a guest in a virtual server environment.

Jörg

On Sat, Mar 14, 2015 at 5:06 PM, Lindsey Poole lpoole@gmail.com wrote:

btw - we're on EC2 I2-4xl hosts, so we have ~120g ram and SSDs.

On Saturday, March 14, 2015 at 9:04:34 AM UTC-7, Lindsey Poole wrote:

I did see the ES_DIRECT_SIZE, but it seems to be ineffective.

I will try setting -XX:MaxDirectMemorySize directly.

On Saturday, March 14, 2015 at 4:43:22 AM UTC-7, Jörg Prante wrote:

You may try limit direct memory on JVM level by
using -XX:MaxDirectMemorySize (default is unlimited). See also
ES_DIRECT_SIZE in http://www.elastic.co/guide/en/elasticsearch/
reference/current/setup-service.html#_linux

I recommend at least 2GB

Jörg

On Sat, Mar 14, 2015 at 1:03 AM, Lindsey Poole lpo...@gmail.com
wrote:

Hey guys,

We're running into some problems under heavy write, nominal read
volume when the Lucene memory mapped files have exhausted available
physical memory, and segments from disk must be paged into memory.

Are there any configs available to control how much physical memory is
available to MemoryMapDirectory?

Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cf50ed46-cacf-414b-8b20-b82595dc2fd0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cf50ed46-cacf-414b-8b20-b82595dc2fd0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHY_M2BAywrG%2BaNcg59xA4_ocph1oqE0bzba4HTqrdLqQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHY_M2BAywrG%2BaNcg59xA4_ocph1oqE0bzba4HTqrdLqQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-%2BAWW-v1XfUZaKvTDDgFxLv3X5gegShk8Uzi1egPa3Aw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Just to close this out - we disabled EC2's health checks and spent some
time tuning the batch thread-pool size to prevent overrunning the cluster
once the memory map cache size exceeds available physical memory. This was
successful (we're restricted to a surprisingly small threadpool size of 3).

Thanks.

On Saturday, March 14, 2015 at 1:05:49 PM UTC-7, Mark Walkom wrote:

Can you provide more info on what the error/problem is, logs might help.

On 14 March 2015 at 10:12, joerg...@gmail.com <javascript:> <
joerg...@gmail.com <javascript:>> wrote:

I'm out - no experience with EC2. I avoid foreign servers at all cost.
Maybe 120G RAM is affected by swap/memory overcommit. Do not forget to
check memlock and memory ballooning. The chances are few you can control
host settings as a guest in a virtual server environment.

Jörg

On Sat, Mar 14, 2015 at 5:06 PM, Lindsey Poole <lpo...@gmail.com
<javascript:>> wrote:

btw - we're on EC2 I2-4xl hosts, so we have ~120g ram and SSDs.

On Saturday, March 14, 2015 at 9:04:34 AM UTC-7, Lindsey Poole wrote:

I did see the ES_DIRECT_SIZE, but it seems to be ineffective.

I will try setting -XX:MaxDirectMemorySize directly.

On Saturday, March 14, 2015 at 4:43:22 AM UTC-7, Jörg Prante wrote:

You may try limit direct memory on JVM level by
using -XX:MaxDirectMemorySize (default is unlimited). See also
ES_DIRECT_SIZE in http://www.elastic.co/guide/en/elasticsearch/
reference/current/setup-service.html#_linux

I recommend at least 2GB

Jörg

On Sat, Mar 14, 2015 at 1:03 AM, Lindsey Poole lpo...@gmail.com
wrote:

Hey guys,

We're running into some problems under heavy write, nominal read
volume when the Lucene memory mapped files have exhausted available
physical memory, and segments from disk must be paged into memory.

Are there any configs available to control how much physical memory
is available to MemoryMapDirectory?

Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/12d0d4ba-8a32-4c85-9635-40d7791865e5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cf50ed46-cacf-414b-8b20-b82595dc2fd0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cf50ed46-cacf-414b-8b20-b82595dc2fd0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHY_M2BAywrG%2BaNcg59xA4_ocph1oqE0bzba4HTqrdLqQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHY_M2BAywrG%2BaNcg59xA4_ocph1oqE0bzba4HTqrdLqQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4bfeb6fa-5fc4-408c-a888-47553ed00738%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.