Elasticsearch Marvel Retention Period?

Elastic Stack Elasticsearch
1

One of the top requests in Marvel came from the community asking for better management of daily indexes. “Marvel is too heavy,” we heard a number of times. In the current release, Marvel now has a default retention duration of 7 days. Beyond this period, the indices created by Marvel will be automatically deleted.

There are subscription licenses available that enable configuration of longer and customized retention intervals. Learn more about Elastic’s subscriptions, which also grant access to security, alerting and the new graph plug-in.

Just got an email today about Marvel index retention.

To me this does not sound like a good reason to put an index retention as default in Marvel free and move the retention customization into subscriptions. They can just use Curator to delete whatever indexes they want. I would be very helpful to keep a history of cluster health for more than the default 7 days.

It's kinda odd to see people submitting for such weird requests.

2

It's simple to workaround this with the free license: use the http exporter. You can point it at the local cluster and it will behave in the same way.

The issue that we see all of the time is that users will run Marvel (1.x or 2.x) using the local exporter, but then do not use something like Curator. As a result, the .marvel-* indices eventually blow up the cluster.

In fact, the best strategy is to point your Marvel exporters (marvel-agent) to an external monitoring cluster for a few reasons:

  • Avoid impacting the production/monitored cluster
  • Allow you to actually monitor the production/monitored cluster even if it's gone

It has the added benefit that it forces you to use the http exporter, which does not have any retention requirements.

3

Thank you for clarifying the changes.