"The field [metricset.name] must be aggregatable across all indices" and a huge rolled-up index

[yagodorea]

I'm using an on-prem version of Elasticsearch in a GCP instance, and I'm having to deal with the huge amounts of data quickly accumulating into my instance's storage. Especially Metricbeat data, that is far to extensive and detailed.

We decided on using the Rollup Jobs feature in Elasticsearch and I'm trying to make the rollup job now, but I'm getting this message when trying to create the job:

The field [metricset.name] must be aggregatable across all indices

It's not just this term, almost every term is displaying this error, which is funny because there's nothing special about the ones that are not (user.name is displaying the error but source.user.name is not).

Also there are always [term] and [term.keyword], the .keyword fields are not displaying the error either, but in my experience with Elasticsearch I can't query those fields, isn't that right? I'm not sure about their purpose.

There's also a second problem. I was making a very simple test rollup job using this very Metricbeat index, taking only one numeric field as metric, taking only the average and the date histogram to 20m (the fine-grained data comes every 10s). After rolling up the 8gb index, my new rolled-up index had 1.8gb! That's way too much for only one field and 20m histogram, right?

What should I do to be able to rollup this index with normal terms and make this rolled up index be smaller?

[yagodorea]

Well now that I understand the rollup feature better, i found out that:

1. You save terms query by marking the '.keyword' field instead of the field itself
2. Histogram fields make the rollup huge and mess up the timestamp (we lose the date field so we can't do much with the data), so you should only save the metrics and terms OR the timestamp, at least while the rollup features don't get better.

Now I ran into some other minor issue that is about data visualization. Rollup indice patterns have many issues in normal visualizations (it's a beta feature after all), and if I use a normal indice pattern my "avg" fields aren't an average of the period, but their sum. I can't work around this with visualizations but I can be aware of this scaling so I can live with it, I guess.

Anyway I'm gonna mark this as resolved!