I use Kibana and X-Pack to monitor my es cluster, but the monitor indices are not automatically deleted. How can I change the config to automatically delete old monitoring-es-logs?
I believe you'd have to set that up like you would to automatically delete any other elasticsearch index, which should be possible with Curator - https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/about.html
@pickypg, can you confirm monitoring doesn't have any built in capabilities for auto deleting old indices?
There are two approaches to X-Pack Monitoring (and Marvel 2.x):
- Local Exporter (default) - this exports data from the cluster back into itself (hence local).
- HTTP Exporter - this exports data from the cluster to a theoretically remote cluster (nothing stops you from pointing it back to the same cluster).
If you use the Local Exporter, then it will clean the indices after 7 days (the 8th day is dropped). If you use the HTTP Exporter, then it will not automatically run the "Cleaner Service" because it doesn't want to assume ownership of all monitoring indices on the remote cluster (it's possible other clusters are sending them there too, and their curation needs may be different).
The best practice is therefore to use the Curator, as @Stacey_Gammon pointed out. However, there is also a workaround if you don't want do that while using the HTTP exporter.
On the remote cluster (aka, the monitoring cluster), install X-Pack and leave Monitoring enabled, but disable data collection for Monitoring:
xpack.monitoring.collection.interval: -1
This will leave the Local Exporter as the default exporter, but it won't actually do anything. However, by enabling the Local Exporter, you will also have the Cleaner Service enabled, which means it will prune indices after the 7th day on that cluster.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.