Rollup to multiple indexes

Elastic Stack Elasticsearch
1

Hi,

We have started using Snowplow to capture usage data on our website and send it to an ES index (called snowplow). Each document in snowplow is large, and currently our historical data reporting only needs two data points - page views, and clicks.

We have created two historical indexes in ES - 'page views' and 'clicks', that are dumps from our MySQL database, and contain monthly aggregates. e.g. { page: '/test', views: { '2016-02': 437, '2016-03': 490 } }.

We could like to run a monthly rollup from snowplow into each of these indices, however the UI in Kibana 6.5.0 only allows us to rollup from one index to another.

Is it advisable to schedule multiple rollup jobs on the same index? Or is this something that's possible in Curator?

Thanks.

2

Not entirely sure I follow, is this correct: you'd like to take the data in snowplow index and roll it up into two separate indices (page_views, clicks), where each index is a different subset of the snowplow metrics?

If that's the case, then yes, I'd go ahead and just create one rollup job for each of those scenarios. It's currently a technical limitation on the Elasticsearch side which dictates the one-to-one mapping of rollup indices.

Well, actually, rollup indices can collect data from source multiple indices (via index patterns like foo-*), but the opposite -- one source index to multiple rollup indices -- is not currently supported.