I have been investigating index federation, and I have a few questions about our options with Elasticsearch.
At a high level, I understand that there are some options for federation.
First, and most basically, we can simply place multiple indices in the same cluster, and then specify the ones we are interested when we run our searches.
More interestingly, the 'Tribe node' feature allows us to federate multiple clusters, but I haven't been able to find a lot of details about it in the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-tribe.html
As I understand it, these features work primarily with an essentially homogeneous document: A search is executed on all indices in tandem, and relevance is used to rank a page of results from each index and return a unified set.
Our use case is distinct from this, though. We have many indices that contain very different information about the same documents, each of which could be maintained by a different group. We would prefer that at least some of the groups be responsible for their own clusters, especially since they may have a large amount of data. Much of the data is very domain specific, and each group knows best how to maintain their portion and may do so with varying degrees of reliability and expertise.
When a search arrives, we need to identify which fields pertain to which indices, and create targeted subqueries to obtain relevant results from each of the other indices, and then aggregate those results together in some manner.
Do any of the features in elasticsearch provide this capability?
If not, are you aware of any solutions that might be used for this approach in general? We are not opposed to looking elsewhere for a solution, but elasticsearch is our platform of choice for many reasons.
Thanks for the help!