I have a largish(?) metricbeat installation which collects from ~2000 machines minutely, generating ~100GB index with 135M docs per day. Currently I have the indexes set to 5 shards x2 replication, which is performing well enough when querying a few days. In this context, I have two questions:
- Would you recommend splitting the daily indexes by metricset to reduce the sizes so that aggregations over longer time periods (>7 days) run more efficiently? If I split by metricset, should I adjust the number of shards?
- What combination of forceMerge/shrink operations would be appropriate to consolidate older metricbeat data? I found this article, but it seems targeted at ES 2.0, is this advice still relevant?
Thanks in advance for any advice you can give!