Old .monitoring indices aren't being deleted

[oblakeerickson (Blake Erickson)]

I'm having a similar issue as this topic where old monitoring indices aren't being deleted:

Deleting Old Monitoring Indices

Hi crickes, The automatic deletion of old monitoring indices is available only for exporters of type local and the retention duration can be configured using the setting xpack.monitoring.history.duration if you have a non-basic license (see https://www.elastic.co/guide/en/x-pack/current/monitoring-settings.html#monitoring-collection-settings). This feature is not available when using http exporters, in this case you can use Curator.

But we are using exporters of type local as far as I know because the monitoring indices are on the same nodes as our other indices. We are using elasticsearch 6 and a basic license.

We are actually running elasticsearch in many different env and this issue is only happening in one of those environments. In other environments the monitoring indices are being deleted after 7 days.

I feel like this elasticsearch cluster in this env got into a weird state somehow since its working fine in all the other environments. Is there something I can check/verify to get this working again? I know I can simply setup curator to run via a cron job because we are already doing this for non-monitoring indices.

[theuntergeek (Aaron Mildenstein)]

I updated the title to reflect that this is exclusive to the .monitoring-* indices.

I don't have any clear insights to why they might not be deleting automatically, unless the settings that enable/disable this have been configured, or a longer-than-7 day retention period is configured.

[oblakeerickson (Blake Erickson)]

@theuntergeek

unless the settings that enable/disable this have been configured, or a longer-than-7 day retention period is configured.

I'm pretty sure with a basic license you can't configure these settings anyways?

@TimV

The default value is 7 days for all licenses.
The ability to configure that setting only exists in Gold or Platinum.

In our elasticsearch.yml we don't have any monitoring related settings set.

[theuntergeek (Aaron Mildenstein)]

I'm not sure what's happened, then.

[oblakeerickson (Blake Erickson)]

Okay I just found another piece of the puzzle. For some reason we are getting a license expired (I'm assuming some x-pack thing?) in the environment and not on the others. But a key line of the license expired elasticsearch logs output is:

The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]

So that is probably what is causing the .monitoring indices to not be deleted. I'll look into the license expired issue and see what I can figure out.