We are investigating to create indexes for a new ES system. From 6.x, ES only supports one type in one index, that means we need to create many indexes, Does that mean new versions of ES has better performance to handle many shards , and is able to handle many many indexes, such as hundreds of or thousands of indexes, which have many shards in total ?
We plan to set up our new system on ES 6.x, the system is to store entity metadata for different apps into ES , then search from ES.
All apps have about 21 common properties , their data are similar, but only one different property called "appOwnData", it is an object type, each app can define its "appOwnData" on its demands. So each app's "appOwnData" should be different. Each app may have about 2 - 10 properties in "appOwnData", There maybe 4 - 10 apps.
The most general search cases are to query something in one app in one org. There are also some cases to search in several apps in one org , or search in several apps in all orgs.
We also plan to generate new index in a new year, such as 2017_app1, 2017_app2, 2018_app1, 2018_app2,
then in future there will be many indexes.
It seems like it is better to create one index for one app because of the "appOwnData", instead of putting all app into one big index.
Here suppose ES cluster nodes can be set up enough many to meet requirement.
I have concern on performance with so many indexes , Any suggestions?