Rollups for beats?

Elastic Stack Beats
1

Now that Elastic Stack 6.3 has shipped with the Rollups API, I'm wondering if there is any plan to develop a set of "default" or jumping-off point rollup configs for Beats. I'm sure many people would like to rollup their beat indices to save space (I'm looking at you, packetbeat). I'm having a bit of a hard time wrapping my head around the best way to configure the rollups, especially with the heterogeneous documents that some beats generate.

Thoughts? Thanks.

2

We definitively have some plans around providing jobs / templates examples that can be reused. Here is a first PR to discuss it: https://github.com/elastic/beats/pull/7220 But it also has some blocker from the Beats perspective as you can see in the PR description.

Would be great to hear from you on what you would expect on how Beats would load rollups and on how you would expect the workflow from a user perspective.

3

Sounds great.

At a minimum, I would like to see some example rollup jobs that I could send to my elasticsearch instance.

I'm new to elasticsearch and I don't want to spend a ton of time making decisions about which fields I need to group on and which metrics to collect, but it would be nice to be able to say "I know I will never need metric x, discard it".

I would also want to customize rollup resolution. My point of reference is graphite, where you can easily configure multiple tiers of data resolution. For example, after 1 day, reduce resolution to 10s, after 7 days reduce to one minute, etc.

It would also be great to put this in the beats config file and have the beat process create the rollup jobs for me.

The default dashboards should be rollup-aware.

4

@christopher-b Thanks a lot for sharing your expected behaviour here. Appreciate it.

5

I too would love to see something like this. There are 1,295 fields in my metricbeat indices and creating the groups and metrics for all of those seems like a monumental and tedious task.

It would be nice to just say please give me 5 minute averages for everything.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.