Slow bulk deletes



I am using elasticsearch 5.6 on a 3 node configuration, lately my bulk delete requests are being processed very very slow, it took almost 2 mins to delete 200 records, my elastic cluster looks fine, nothing specific in error logs, any suggestions on what it could be ?

index settings: shards: 50, refresh interval 5sec, translog durability async

(Mark Walkom) #2

How many indices, shards, docs.


on cluster: around 4000 shards, 80 indices, 3 nodes, master is also a data node

(Mark Walkom) #4

That's a lot of shards for that many indices and nodes.

How many indices are you deleting from with that 200 docs?


the entire bulk request only has delete operations all on the same index

(Mark Walkom) #6

How much data does the cluster hold?


300GB in total, used approx 25%

(Mark Walkom) #8

Ok, way too many shards then. You need to reduce that amount and you should find things will speed up.


sure will try it out. thank you! @warkolm . btw how many shards would you suggest per index in this configuration, any standard ? like better to have these many shards if you have these many nodes ?

i have another configuration with 14 nodes and 3 masters and 11 data nodes

(Mark Walkom) #10

It really depends on your use case, but I would be aiming for 30-50GB per shard.


okay, so since i only have 300gb which is 100 gb per node, so I would only have 10 shards for the entire cluster ?

also I have 4000 on the entire cluster, for the specific index an dealing with, it has 50 shards

(Mark Walkom) #12

How large is that index?


one replica, so shards 50 X 2
docs 100k, size 350 mb

(Mark Walkom) #14

That's waaaaay too many shards.

(system) closed #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.