ETL/complex data transformations

Elastic Stack Kibana
1

Hi,

Is it possible or a good pattern to have complex data transformations (ETL like) running behind a visualization/dashboard based on an single index present in Elasticsearch? I would like to understand the data modeling best practices for index if there are reporting needs based on the same data source but different pattern of aggregations and filters (usually these transformations consisting of multi-step business logic).
(a) If there are complex multi-step filters and aggregations required to answer certain reporting questions of a visualization, would it be advisable to create separate and already transformed indices along with the main source index?
(b) Or should I keep just that single main index and try to code the logic of transformation to run during run time using tools such as Vega? Is it a good idea to use Vega to run complex multi-step transformations required for the reporting?

In other words, when should we start thinking about creating new smaller pre-transformed (ETL) indices instead of just keeping that single original index.

Thanks

2

I don't know of a specific pattern here, so it's probably do what you think is best. I would definitely look to put things into their own index, it maintains data integrity and keeps things like mapping and index management a little easier.

3

Thanks.

One followup question. Can Vega produce tabular output as visualization, similar to "datatable" visualization?

4

I don't know enough of Vega sorry, you'd be better off making a new topic for it with the Vega tag, so an expert can help :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.