I've been using ES in production for a long time for our application, but it is currently single tenant, and so I have dedicated ES instances running for each client. We are moving towards a multi-tenant platform and I just want to make sure I'm approaching this well from a hardware/config perspective.
Currently, I'm storing a total of roughly 7TB of data across all these tenants. The plan is to move this all into a multi-tenant system. This could all be in one huge cluster, or we could separate customers per cluster - it's not too tricky either way. Looking for best recommendations here to support future growth as well.
For queries and indexing, we generate a
routing key from each tenant to force their data to a specific shard. I'm currently planning on having three master nodes, and I'm having a hard time finding a good recommendation on hardware sizing here. Am I right in thinking that dedicated master nodes don't need a ton of resources compared to data nodes?
For the data nodes:
- Based on 7TB of data (and growing) what would you recommend as a quantity of data nodes and hardware specs? I'm assuming many boxes, each with 64GB of RAM and a <32GB heap is optimal, but is there a point that it is more sensible to have multiple independent clusters rather than trying to scale one cluster out?
- What are the critical statistics to monitor to know when we should add another data node?
- We currently have one index per object type in our application (e.g. all users are in a
usersindex, all payments are in a
paymentsindex, etc. Just want to make sure that's the best approach - in our legacy app, we have an
entitiesindex with each object being a separate type, but it appears that is not recommended anymore.
- Based on this, I feel like we should have a relatively low shard count, as each index doesn't have a ton of data in it (we probably have 100+ indexes at this point.) That being said, with it being multi-tenant now, it will be multiplied by the number of customers. Thoughts?
Anything else I should bear in mind in this transition? Appreciate any feedback!