Identifying the stale indices in Elasticsearch

[vineethsuhas (Vineeth Suhas Challagali)]

Hi,

I want to delete the indices which are stale (Not being accessed since last month). Is there a way to identify the indices which are not in use since one month?

I could not find any stats on the indices which corresponds to the last_accessed_time (Either read/write).

Just wondering if the timestamp is not maintained, but is there a way to monitor the elastic request logs to monitor the indices and delete the indices which are not used in the past month? If this is a possible solution, how to access the elastic request logs in the AWS Elasticsearch service?

Any suggestions on solving this problem would be much appreciated!

Best,
Vineeth

[dadoonet (David Pilato)]

Welcome!

@vineethsuhas

how to access the elastic request logs in the AWS Elasticsearch service?

We don't support this service. Did you look at Cloud by Elastic, also available if needed from AWS Marketplace, Azure Marketplace and Google Cloud Marketplace?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named Observability, Security, Enterprise Search and what is coming next ...

@vineethsuhas

Any suggestions on solving this problem would be much appreciated!

Anyway, I don't think there's such a thing unless you activate the audit logs but this feature is not available on the service you are using I think. See Enable audit logging | Elasticsearch Guide [8.11] | Elastic

[vineethsuhas (Vineeth Suhas Challagali)]

Hello @dadoonet.

Thank you for the response.

I am glad to know about the Cloud by Elastic. Although it is helpful to have a completely managed Elasticsearch service, I am looking forward to the quick solution which could help me maintain my environments clean and optimize the space utilization.

Its very unfortunate to see that we are maintaining the number_of_queries on the index in the _stats but not the time of the latest query. Hope to see it coming in the future releases of the Elasticsearch

Best,
Vineeth

[dadoonet (David Pilato)]

@vineethsuhas

Hope to see it coming in the future releases of the Elasticsearch

Even if this comes in a coming version, you might not have an access to this feature as long as you are using a 3rd party service and not the official one.

That being said, how do the end users access to elasticsearch? Are they using Kibana or your own application?

[vineethsuhas (Vineeth Suhas Challagali)]

@dadoonet The end users use Elasticsearch through our application.

I am mostly concerned about the dev/test environments where we can create the indices in an unregulated fashion which could lead to some space utilization issues in the longer run.

I am interested in an additional statistic which can be retrieved using index/_stat to provide an additional last_accessed_time statistic which can help me in detecting the unused indices and delete them.