Slow bulk deletes


#1

Hello

I am using elasticsearch 5.6 on a 3 node configuration, lately my bulk delete requests are being processed very very slow, it took almost 2 mins to delete 200 records, my elastic cluster looks fine, nothing specific in error logs, any suggestions on what it could be ?

index settings: shards: 50, refresh interval 5sec, translog durability async


(Mark Walkom) #2

How many indices, shards, docs.


#3

on cluster: around 4000 shards, 80 indices, 3 nodes, master is also a data node


(Mark Walkom) #4

That's a lot of shards for that many indices and nodes.

How many indices are you deleting from with that 200 docs?


#5

the entire bulk request only has delete operations all on the same index


(Mark Walkom) #6

How much data does the cluster hold?


#7

300GB in total, used approx 25%


(Mark Walkom) #8

Ok, way too many shards then. You need to reduce that amount and you should find things will speed up.


#9

sure will try it out. thank you! @warkolm . btw how many shards would you suggest per index in this configuration, any standard ? like better to have these many shards if you have these many nodes ?

i have another configuration with 14 nodes and 3 masters and 11 data nodes


(Mark Walkom) #10

It really depends on your use case, but I would be aiming for 30-50GB per shard.


#11

okay, so since i only have 300gb which is 100 gb per node, so I would only have 10 shards for the entire cluster ?

also I have 4000 on the entire cluster, for the specific index an dealing with, it has 50 shards


(Mark Walkom) #12

How large is that index?


#13

one replica, so shards 50 X 2
docs 100k, size 350 mb


(Mark Walkom) #14

That's waaaaay too many shards.