Indexing Buffer Settings per Index

Elastic Stack Elasticsearch
1

Context - I want to increase the performance of indexing. We have implemented a logstash kind of architecture where the indices are split per week. So at a time we have some indices where we write data but we have a lot of indices where we don't index but search only.

The usual question is "What optimization can we do?"

I got some idea from https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing
There is a clause there where we can increase the indices.memory.index_buffer_size but it is a node level setting.

Is there a way to manually override the above setting for only the indices being written? I want to increase the buffer size for some indices but keep it as default for other indices.

2

I found this great forum thread about tuning index buffer size and how it is distributed amongs active shards. This suggests that you don't have to optimize this setting for the index that is beeing written, it will just use the buffer for shards that are active (i.e. beeing written to).

Another thing to be aware of for your use case might be the shrinking api that is coming up as a new feature in the soon-to-be-releases 5.0 version of Elasticsearch. This will allow you to basically use a higher shard number for the indices that are actively written to while reducing the shard count for the ones that are only read from. This blog post explains it in a little bit more detail, but mind you this is a feature thats not available in the 2.x versions of ES yet.

1 Like
3

thanks man. This is what I needed.