Speed up shard allocation on Elasticsearch 6.x single node cluster

I'm currently upgrading a "cluster" (with a single node) from Elasticsearch 1.x to 6.x. Everything is working out great, but I experience very slow shard allocation on 6.x. I don't remember if I did anything on the old cluster to speed things up. But on a single node cluster, shard allocation should be pretty straightforward.

Is shard allocation slower on 6.x or can I look through some settings on my 1.x node to see what is different?

Indeed it should be. How many shards do you have in your cluster?

I have >5,500 shards spread across around 1,800 indices. I totally understand that allocation takes some time with this amount of shards. The only thing I'm wondering about, is the performance difference between the two versions.

This is far too many, and you will definitely see performance issues as a consequence. You should try and reduce this number. Here is an article about shard counts:

The 1.x series was a long time ago, and modern Elasticsearch does many things quite differently. I do not find it surprising that there's a performance difference in this area.

1 Like

Thank you for the link to that blog post. Would love to have seen that earlier :slight_smile: I save log data for customers where a customer can have multiple logs (each log can store multiple log messages). Each log is represented by an index consisting of three shards. From what I can see in the article, I should probably have an index per customer, rather than per log per customer.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.