Speed up shard allocation on Elasticsearch 6.x single node cluster

Elastic Stack Elasticsearch
1

I'm currently upgrading a "cluster" (with a single node) from Elasticsearch 1.x to 6.x. Everything is working out great, but I experience very slow shard allocation on 6.x. I don't remember if I did anything on the old cluster to speed things up. But on a single node cluster, shard allocation should be pretty straightforward.

Is shard allocation slower on 6.x or can I look through some settings on my 1.x node to see what is different?

2

Indeed it should be. How many shards do you have in your cluster?

3

I have >5,500 shards spread across around 1,800 indices. I totally understand that allocation takes some time with this amount of shards. The only thing I'm wondering about, is the performance difference between the two versions.

4

This is far too many, and you will definitely see performance issues as a consequence. You should try and reduce this number. Here is an article about shard counts:

The 1.x series was a long time ago, and modern Elasticsearch does many things quite differently. I do not find it surprising that there's a performance difference in this area.

1 Like
5

Thank you for the link to that blog post. Would love to have seen that earlier :slight_smile: I save log data for customers where a customer can have multiple logs (each log can store multiple log messages). Each log is represented by an index consisting of three shards. From what I can see in the article, I should probably have an index per customer, rather than per log per customer.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.