Shards Bulk exception

Elastic Stack Elasticsearch
1

See loads of bulk exceptions .... could it be due to high number of shards ...

we have 24 data nodes 10,800 shards ? and discovery.zen.minimum_master_nodes = 13

queue size = 410

let me know ?

2

Are you seeing exceptions or rejections? How many of these shards are you actively indexing into?

3

Rejections ... almost most of the shards are actively indexing .....

4

Then please have a look at this blog post. Reducing shard count or making each bulk request target only a few indices might help reduce the amount of rejections.

5

Thanks Chris. Just read the blog and wondering since the cluster seems well balanced and shards are equally spread out.

is queueSize 410 good for 24 data nodes ? Reading the documents i see it is less and can be upped a bit to 600. Appreciate your valuable suggestions on this regard ..

6

Increasing the queue size leads to increased resource usage and does not necessarily solve the underlying problem. It is more like putting a band-aid on.

I would recommend looking into how you are indexing and see if there are improvements to be made there or by changing the sharing scheme.

How many indices do these shards belong to? How much data do they hold?

7

440 shards to each node and average size of indices on each node is around 280GB

8

That gives an average shard size of less than 1GB which is very small. Have a look at this blog post for some practical guidelines around shards and sharding.

9

Thanks Chris. I will go through this.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.