# Tuning : index\_buffer\_size and active shards

**URL:** <https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576>\
**Category:** Elasticsearch\
**Created:** [December 7, 2015, 9:05pm UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576 "2015-12-07T21:05:45Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bruno\_Lavoie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruno_lavoie/32/8408_2.png) [@Bruno\_Lavoie](https://discuss.elastic.co/u/Bruno_Lavoie)\
**Post date:** [December 7, 2015, 9:05pm UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/1 "2015-12-07T21:05:45Z")

</div>

Hello,

We're going to put in place a write heavy cluster for logs storage using logstash.

So, I saw that it's recommended to increase the indexing buffer. But I'm a bit puzzled about the fact that this space is divided between all shards on a node.

I don't remember where, but I saw somewhere that this space is divided between _active shards_, where an _active shard_ is one with at least one indexing operation in the last 5 minutes or so.

Depending on the validity of this, it's quite a gamechanger for my calculations.

Given the fact that, we have about 10 different logs sources (about to increase) with each an index per day, with 5 shards and 2 replicas each, for 60 days retention. This gives a quite big number of shards to maintain and to split this memory on our 5 nodes.

For short, the true questions are:

- What is an active shard?
- Are old indices, 1 days ago and latter, consuming space of the indexig buffer space?

I hope that :

- _active shards_ are those with an indexing activity in the last X minutes.
- indexing buffer space is really divided between those

It would make sense....

In the same settings. If we specify indices.memory.min\_shard\_index\_buffer\_size:

- is it a hard minimum of memory given to all shards? including those _inactive_?
- if we have a lot of shards, can this sum bust something like heap?

Specs:  
Our estimates is to have 10k -20k events per seconds on a 5 nodes cluster.  
Starting with 16 GB of heap on a 64 GB server, about to increase to 28 GB if our benchmarks suggest that.

Thanks  
Bruno Lavoie

---

<div class="post-metadata">

**Author:** ![Bruno\_Lavoie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruno_lavoie/32/8408_2.png) [@Bruno\_Lavoie](https://discuss.elastic.co/u/Bruno_Lavoie)\
**Post date:** [December 8, 2015, 4:40pm UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/2 "2015-12-08T16:40:33Z")

</div>

Good, response from me to me, and to help others.

Thanks to open source! Without being a programmer, I looked to source code.

The facts are:

- The indexing buffer is divided between **actively indexing shards** on the node.

- An **actively indexing shard** , is one with an activity in the last 5 minutes.

- The check classifying actively/inactively indexing shards is performed each 30 seconds.

- Even if you allocate a big numer to this buffer, there is a hard limit of 512 MB of memory for each shards. Becasue beyond that value, tests proved that it doesn't increase indexing performance.

- When not actively indexing, shards indexing buffer is set to 500 KB.

Maybe the doc should be updated to be more explicit on these facts.

Main read files:  
`https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/indices/memory/IndexingMemoryController.java`

`https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/index/shard/IndexShard.java`

Bruno Lavoie

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [December 8, 2015, 5:00pm UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/3 "2015-12-08T17:00:17Z")

</div>

Sorry we didn't get back to you earlier! There is some work actively going on in this area and it'll just get better.

> [@Bruno\_Lavoie](#):
>
> Thanks to open source! Without being a programmer, I looked to source code.

If you are willing to sign the CLA then you too can send a documentation update! If you click on the edit link on the page it'll take you to an editor where you can make the change. Or, rather, it should. I promise someone will review it! I can't promise it'll be me because my review queue (open tabs with PRs in them) is pretty deep right now, but someone will!

BTW, you're about 65% of the way to being a programmer now that you can read that source code and make sense out of it.

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [December 8, 2015, 5:03pm UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/4 "2015-12-08T17:03:32Z")

</div>

> [@Bruno\_Lavoie](#):
>
> I don't remember where, but I saw somewhere that this space is divided between active shards, where an active shard is one with at least one indexing operation in the last 5 minutes or so.

This is correct. The setting that controls this is `index.shard.inactive_time` and the default is indeed five minutes.

> [@Bruno\_Lavoie](#):
>
> What is an active shard?

A shard that has received an indexing operation (index or delete) within the last `index.shard.inactive_time` time units.

> [@Bruno\_Lavoie](#):
>
> Are old indices, 1 days ago and latter, consuming space of the indexig buffer space?

Their indexing buffers are idled down to 500kb. This buffer does not count against the global indexing buffer limit. There is a check every `indices.memory.interval` time units (defaults to 30 seconds) on the state of all shards, and a check whenever a shard changes from inactive to active. When these checks are executed, shards that have had a state change from active to inactive will be idled down to this 500kb limit, and then the global indexing buffer will be allocated amongst the remaining inactive shards.

> [@Bruno\_Lavoie](#):
>
> active shards are those with an indexing activity in the last X minutes.

Yes. This is five minutes by default.

> [@Bruno\_Lavoie](#):
>
> indexing buffer space is really divided between those

Yes, only the active shards.

> [@Bruno\_Lavoie](#):
>
> is it a hard minimum of memory given to all shards? including those inactive?

Inactive shard are always idled down to 500kb. The `indices.memory.min_shard_index_buffer_size` setting is only for active shards.

> [@Bruno\_Lavoie](#):
>
> if we have a lot of shards, can this sum bust something like heap?

No. There is global indexing buffer size of `indices.memory.index_buffer_size` (defaults to 10% of the heap) that is apportioned equally to the active shards. This means that even if you set the maximum to 512 MB, if your heap is 1g, then no single shard will have an indexing buffer large than 102.4 MB.

---

<div class="post-metadata">

**Author:** ![Bruno\_Lavoie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruno_lavoie/32/8408_2.png) [@Bruno\_Lavoie](https://discuss.elastic.co/u/Bruno_Lavoie)\
**Post date:** [December 8, 2015, 5:06pm UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/5 "2015-12-08T17:06:24Z")

</div>

Thanks for your reply, better late than never.

I'm willing to make documentation update, but english is not my native language. I'm an ex-programme, now in the world of databases and analytics. Elasticsearch is simply awesome, but I prefer to understand each details of a datastore before using it in production., and get as ready as possible when troubleshooting is necessary.

Hope that my conclusions aren't bad on theses settings.

Maybe I'll make a pull request in a few days.  
Hope to help the community.

---

<div class="post-metadata">

**Author:** ![Bruno\_Lavoie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruno_lavoie/32/8408_2.png) [@Bruno\_Lavoie](https://discuss.elastic.co/u/Bruno_Lavoie)\
**Post date:** [December 9, 2015, 2:14am UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/6 "2015-12-09T02:14:40Z")

</div>

Thanks, this makes everything clearer.  
Excepted one point regarding min\_shard\_index\_buffer\_size.

> [@Bruno\_Lavoie](#):
>
> In the same settings. If we specify indices.memory.min\_shard\_index\_buffer\_size:
> 
> - is it a hard minimum of memory given to all shards? including those inactive?
> - if we have a lot of shards, can this sum bust something like heap?

Well, for the first point, I understand that it's only for actively indexing shards.

For the second, you responded:

> [@jasontedor](#):
>
> No. There is global indexing buffer size of indices.memory.index\_buffer\_size (defaults to 10% of the heap) that is apportioned equally to the active shards. This means that even if you set the maximum to 512 MB, if your heap is 1g, then no single shard will have an indexing buffer large than 102.4 MB.

Maybe my wording was bad, but my question was about the minimum indexing buffer for each shards. Even with the default 4 MB setting, a huge number of actively indexing shards can exceed and bypass the established index\_buffer\_size (default 10% of the heap).

Maybe I'm missing something, but this exerpt from IndexingMemoryController doesn't seem to enfore any hard upper limit.

My comments with //@@@@.

```
int activeShardCount = activeShards.size();

[...]

if (activeShardCount == 0) {
    return;
}

//@@@@ split indexing buffer space evenly between active shards
ByteSizeValue shardIndexingBufferSize = new ByteSizeValue(indexingBuffer.bytes() / activeShardCount);

//@@@@ if the split is inferior to requested minimum (default 4MB), the requested value takes precedence
if (shardIndexingBufferSize.bytes() < minShardIndexBufferSize.bytes()) {
    shardIndexingBufferSize = minShardIndexBufferSize;
}

//@@@@ but always enforce the hard maximum of 512 MB
if (shardIndexingBufferSize.bytes() > maxShardIndexBufferSize.bytes()) {
    shardIndexingBufferSize = maxShardIndexBufferSize;
}

[...]

//@@@@ apply established buffer size
for (IndexShard shard : activeShards) {
    updateShardBuffers(shard, shardIndexingBufferSize, shardTranslogBufferSize);
}

```

To recall your example:

- with the case of a 1 GB heap
- with a 102.4 MB (10% default) indexing buffer
- with default minimum of 4 MB per shard

With my understanding of the source code, a node with 50 actively indexing shards will allocate 200 MB, roughly twice than the 102.4 MB (10%). And this can grow up without any hard hard limits, say 80% of the heap.

Maybe it's a bad example or extreme case, because most of the time, heaps are way larger. But, I really want to understand all the interractions between these parameters.

So, we just need to be cautious about this fact.  
Especially if we have a multi-tenant cluster with a lot of active clients.

Is there:

- Any way to monitor the current indexing buffer allocated space?
- Any way to show current actively indexing shards on a node, with their respective buffer size?

If yes, I'll use it to send alerts to our monitoring systems (in extreme cases).  
All this, even if we do good planing.

And, are the inactive 500 KB shards indexing buffer accounted in the indexing buffer?

Thanks for your precious time.  
Bruno Lavoie

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [December 9, 2015, 2:58am UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/7 "2015-12-09T02:58:25Z")

</div>

> [@Bruno\_Lavoie](#):
>
> Maybe my wording was bad, but my question was about the minimum indexing buffer for each shards.

The fault lies with me and I apologize for misunderstanding this question.

> [@Bruno\_Lavoie](#):
>
> Even with the default 4 MB setting, a huge number of actively indexing shards can exceed and bypass the established index\_buffer\_size (default 10% of the heap).

You are correct.

> [@Bruno\_Lavoie](#):
>
> Maybe it's a bad example or extreme case, because most of the time, heaps are way larger.

Yes, that's part of why this isn't a risky problem. For this to be a problem, the minimum indexing buffer either needs to be set large (relative to the global indexing buffer budget) or there needs to be an incredibly large number of _active_ shards. I would classify both of these as operations mistakes that should just be avoided as they will cause problems in general. But in addition to those two factors, a sufficient number of shards need to be continuously indexing documents (lest those allocated buffers will not fill up, and what is used will eventually be flushed and idled down).

That is, with a sufficiently large minimum buffer size or a sufficiently large number of _active_ shards, the allocated budget can exceed the global indexing buffer budget, but to spill the heap these budgets need to also be in use. This is not a set of circumstances that is at all likely with proper operations management. Let me know if that explanation does not make sense.

> [@Bruno\_Lavoie](#):
>
> Any way to monitor the current indexing buffer allocated space?

These metrics are not exposed through the API, but the indexing buffer reallocations are logged (DEBUG logging level, `indices.memory`).

> [@Bruno\_Lavoie](#):
>
> Any way to show current actively indexing shards on a node, with their respective buffer size?

These metrics are also not exposed through the API, but are again available in the logs (DEBUG logging level, `indices.memory`) and when a shard is idled (DEBUG logging level, `index.shard`).

> [@Bruno\_Lavoie](#):
>
> And, are the inactive 500 KB shards indexing buffer accounted in the indexing buffer?

No, they do not count.

> [@Bruno\_Lavoie](#):
>
> Thanks for your precious time.

Sincerely, thank you for your thoughtful questions and your interest in Elasticsearch. 🙂

---

<div class="post-metadata">

**Author:** ![Bruno\_Lavoie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruno_lavoie/32/8408_2.png) [@Bruno\_Lavoie](https://discuss.elastic.co/u/Bruno_Lavoie)\
**Post date:** [December 9, 2015, 12:29pm UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/8 "2015-12-09T12:29:37Z")

</div>

> [@jasontedor](#):
>
> but to spill the heap these budgets need to also be in use.

Thanks again

IIRC allocation memory on an active shard is à hard limit and this memory is not locked entirely to the shard buffer. Only very active ones that needs to, they will use all allocated budget.

Well done because not all indices have the same incomming throughput.

Definitively, budget is the best word.

Thanks you  
Bruno Lavoie

---

<div class="post-metadata">

**Author:** ![itplayer](https://avatars.discourse-cdn.com/v4/letter/i/d26b3c/32.png) [@itplayer](https://discuss.elastic.co/u/itplayer)\
**Post date:** [January 10, 2017, 2:44am UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/9 "2017-01-10T02:44:58Z")

</div>

Is there anyway to know current number of active shard through REST API?  
I've read that /\_cluster/health?pretty  
Has an "active\_shards" value, but I guess this is not.  
Any idea?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:04pm UTC](https://discuss.elastic.co/t/tuning-index-buffer-size-and-active-shards/36576/10 "2017-07-05T22:04:23Z")

</div>