Slow bulk deletes


I am using elasticsearch 5.6 on a 3 node configuration, lately my bulk delete requests are being processed very very slow, it took almost 2 mins to delete 200 records, my elastic cluster looks fine, nothing specific in error logs, any suggestions on what it could be ?

index settings: shards: 50, refresh interval 5sec, translog durability async

How many indices, shards, docs.

on cluster: around 4000 shards, 80 indices, 3 nodes, master is also a data node

That's a lot of shards for that many indices and nodes.

How many indices are you deleting from with that 200 docs?

the entire bulk request only has delete operations all on the same index

How much data does the cluster hold?

300GB in total, used approx 25%

Ok, way too many shards then. You need to reduce that amount and you should find things will speed up.

sure will try it out. thank you! @warkolm . btw how many shards would you suggest per index in this configuration, any standard ? like better to have these many shards if you have these many nodes ?

i have another configuration with 14 nodes and 3 masters and 11 data nodes

It really depends on your use case, but I would be aiming for 30-50GB per shard.

okay, so since i only have 300gb which is 100 gb per node, so I would only have 10 shards for the entire cluster ?

also I have 4000 on the entire cluster, for the specific index an dealing with, it has 50 shards

How large is that index?

one replica, so shards 50 X 2
docs 100k, size 350 mb

That's waaaaay too many shards.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.