Is it possible or a good pattern to have complex data transformations (ETL like) running behind a visualization/dashboard based on an single index present in Elasticsearch? I would like to understand the data modeling best practices for index if there are reporting needs based on the same data source but different pattern of aggregations and filters (usually these transformations consisting of multi-step business logic).
(a) If there are complex multi-step filters and aggregations required to answer certain reporting questions of a visualization, would it be advisable to create separate and already transformed indices along with the main source index?
(b) Or should I keep just that single main index and try to code the logic of transformation to run during run time using tools such as Vega? Is it a good idea to use Vega to run complex multi-step transformations required for the reporting?
In other words, when should we start thinking about creating new smaller pre-transformed (ETL) indices instead of just keeping that single original index.