Working on old Indices

Elastic Stack Elasticsearch
1

Hi!
I was wondering if it's possible to "thin" documents in an index and by thinning I mean dropping 99.9% of docs in it - let's say all fields are numeric so I would like to find average of 1000 values, remove all those values and end up only with averages of every field.

Reason behind that would be: trying to store 1-2 years of data, around 100GB daily gives something close to 30 TB per year and that's quite too much for my infrastructure. Currently another solution is being used and they are literally storing 1 year of data on a 100 GB filesystem because they are calculating those averages / means for all field so instead of holding 100GB daily they store like 1 MB :slight_smile:
Would that be possible in elasticsearch? Also, I'm using Elasticsearch + Graylog.

2

You mean something like Rollup API?

1 Like
3

Yeah that's exactly the thing I was looking for! Thanks :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.