Hi guys, I have a problem of one shard contain too many docs. So I have an indice, docs in this indice is routed by some kind of an organization_id, so we know that all docs from an organization would be in one shard This lead into a problem that, there're an organization that have too many docs compare to others => one shard is too big while others is small. I'm having a shard has around 60gb docs and others is just 1-2 gb.
So I'm looking for a solution to balance those shards, because sooner, we would hit the limit of shard size.
My current indice has join (parent child relationship) field and nested field, es 6.3
I cant let the system auto routing for me because the parent could be in one shard, the the child have to follow the parent route, (child can be assigned to parent after created). So I have "same" docs in 2 shards if this case happened: child routed to X, parent routed to Y, child assigned to parent, we need to delete child in X and create child in Y
I can think of:
-
Create another indice, this indice is only for one/ or two big organization, all the mapping would be the same. Is this approach scalable and make sense, any cons for this?
-
I can reindex my current structure into none parent-child type (but nested type) and let the system auto do the routing. This would cause edit action a bit more stress than before.
Any other ways around? Thanks!