Indices.memory.index_buffer_size

Elastic Stack Elasticsearch
1

Hi,

Is the "indices.memory.index_buffer_size" configuration a cluster wide
configuration or per node configuration? Do I need to set it on every node?
Or just the master (eligible) node?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f67e3a30-521c-4c13-8620-c79133cea01c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

states "It is a global setting that bubbles down to all the different
shards allocated on a specific node."

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 August 2014 03:12, Yongtao You yongtao.you@gmail.com wrote:

Hi,

Is the "indices.memory.index_buffer_size" configuration a cluster wide
configuration or per node configuration? Do I need to set it on every node?
Or just the master (eligible) node?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f67e3a30-521c-4c13-8620-c79133cea01c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f67e3a30-521c-4c13-8620-c79133cea01c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZrhCzaXc4qAHeiOH8RKr6bKB-oRGH8YDFpz%3D34m29Y3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks Mark.

What confuses me are "global setting" (which suggests cluster-wide setting)
and "on a specific node" (which suggests node level setting). I could just
try it out, but it's hard to tell if the setting worked or not. :frowning:

On Sunday, August 24, 2014 3:13:17 PM UTC-7, Mark Walkom wrote:

Elasticsearch Platform — Find real-time answers at scale | Elastic
states "It is a global setting that bubbles down to all the different
shards allocated on a specific node."

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 25 August 2014 03:12, Yongtao You <yongt...@gmail.com <javascript:>>
wrote:

Hi,

Is the "indices.memory.index_buffer_size" configuration a cluster wide
configuration or per node configuration? Do I need to set it on every node?
Or just the master (eligible) node?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f67e3a30-521c-4c13-8620-c79133cea01c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f67e3a30-521c-4c13-8620-c79133cea01c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d76f4c67-9250-4ab9-b02d-0f0c78b33be6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

I just looked at this code!

Its a setting that you set globally at the cluster level. It takes effect
per node. What that means is that for every "active" shard on each the
node gets an equal share of that much space. "Active" means has been
written to in the past six minutes or so. When a node first starts all
shards are active assumed active and those that are not updated at all lose
active status after the timeout. You can watch the little dance it does by
setting
index.engine.internal: DEBUG
in logging.yml.

Now - I'm not actually sure how important a setting it it is. I opened
indices.memory.index_buffer_size is spread evenly across active shards · Issue #7441 · elastic/elasticsearch · GitHub to get suggest
allowing better spreading it around. Mike'll probably close it if
spreading it around wouldn't really help things much.

Nik

On Tue, Aug 26, 2014 at 2:07 PM, Yongtao You yongtao.you@gmail.com wrote:

Thanks Mark.

What confuses me are "global setting" (which suggests cluster-wide
setting) and "on a specific node" (which suggests node level setting). I
could just try it out, but it's hard to tell if the setting worked or not.
:frowning:

On Sunday, August 24, 2014 3:13:17 PM UTC-7, Mark Walkom wrote:

Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/modules-indices.html states "It is a global setting
that bubbles down to all the different shards allocated on a specific node."

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 25 August 2014 03:12, Yongtao You yongt...@gmail.com wrote:

Hi,

Is the "indices.memory.index_buffer_size" configuration a cluster wide
configuration or per node configuration? Do I need to set it on every node?
Or just the master (eligible) node?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/f67e3a30-521c-4c13-8620-c79133cea01c%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f67e3a30-521c-4c13-8620-c79133cea01c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d76f4c67-9250-4ab9-b02d-0f0c78b33be6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d76f4c67-9250-4ab9-b02d-0f0c78b33be6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1CmkjPAPJns3PjCmsFicu8KYV0DRjv9T2qacx636sy7g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

See also Add `segments.index_writer_max_memory` to stats by mikemccand · Pull Request #7440 · elastic/elasticsearch · GitHub (will be
in 1.4.0) which returns the actual RAM buffer size assigned to that shard
by the "little dance".

Mike McCandless

http://blog.mikemccandless.com

On Tue, Aug 26, 2014 at 2:15 PM, Nikolas Everett nik9000@gmail.com wrote:

I just looked at this code!

Its a setting that you set globally at the cluster level. It takes effect
per node. What that means is that for every "active" shard on each the
node gets an equal share of that much space. "Active" means has been
written to in the past six minutes or so. When a node first starts all
shards are active assumed active and those that are not updated at all lose
active status after the timeout. You can watch the little dance it does by
setting
index.engine.internal: DEBUG
in logging.yml.

Now - I'm not actually sure how important a setting it it is. I opened
indices.memory.index_buffer_size is spread evenly across active shards · Issue #7441 · elastic/elasticsearch · GitHub to get suggest
allowing better spreading it around. Mike'll probably close it if
spreading it around wouldn't really help things much.

Nik

On Tue, Aug 26, 2014 at 2:07 PM, Yongtao You yongtao.you@gmail.com
wrote:

Thanks Mark.

What confuses me are "global setting" (which suggests cluster-wide
setting) and "on a specific node" (which suggests node level setting). I
could just try it out, but it's hard to tell if the setting worked or not.
:frowning:

On Sunday, August 24, 2014 3:13:17 PM UTC-7, Mark Walkom wrote:

Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/modules-indices.html states "It is a global setting
that bubbles down to all the different shards allocated on a specific node."

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 25 August 2014 03:12, Yongtao You yongt...@gmail.com wrote:

Hi,

Is the "indices.memory.index_buffer_size" configuration a cluster wide
configuration or per node configuration? Do I need to set it on every node?
Or just the master (eligible) node?

Thanks.
Yongtao

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/f67e3a30-521c-4c13-8620-c79133cea01c%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f67e3a30-521c-4c13-8620-c79133cea01c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d76f4c67-9250-4ab9-b02d-0f0c78b33be6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d76f4c67-9250-4ab9-b02d-0f0c78b33be6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1CmkjPAPJns3PjCmsFicu8KYV0DRjv9T2qacx636sy7g%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1CmkjPAPJns3PjCmsFicu8KYV0DRjv9T2qacx636sy7g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAD7smRcwFh-qCtu6B15Ni9KjzCYVojXtc4KTzMc%2Be1BVHZ%3D-Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.