Hi @sebastianboelling , thanks for your interest in scaling Enterprise Search!
I'm not aware of an aggregated document on running Enterprise Search at scale, but I'll pass that suggestion along to our team.
In the meantime, I can point you to a few resources that you may find useful:
- In 8.0+, when you create an Engine, you can override the default number of shards and specify a larger number of shards.
- You may also be interested in dynamic index settings such as
index.auto_expand_replicas, and controlling hot spots through settings like index.routing.allocation.total_shards_per_node. - For App Search, I believe you'll be interested in meta engines, which allow you to create an engine built off of multiple other engines - this is often preferable to having a single monolithic engine.
- For Workplace search, you can partition the data for each Workplace Search connector using multiple service accounts/content sources.
Good luck with your scaling!