Deleting old records from ES

Elastic Stack Elasticsearch
1

Hi,

I would what to know how to rotate old records from ES index, coz when i
delete documents with delete query API it does not free up disk space. Eg.
i want to remove records older than 3 days in my index. For me disk space
is an issue.

Thanks and Regards,
Abrar.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/61d35912-72ad-4139-aec3-46fc89f36ac9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi Abrar,

When you run delete by query, documents aren't immediately deleted. They
are marked as deleted and the data gets removed in the process of merging,
which runs asynchronously in the background. There are some settings that
make merging more "sensitive" to deletes, have a look here:

Alternatively, you can force a merge by running optimize:

But this is also problematic because it will thrash your I/O and invalidate
much of your caches.

Usually, a better option if you have expiring data is to have time-based
indices (an index per day, for example). You can then use something like
the Elasticsearch Curator to expire entire indices as the days go by:

Here's a nice blog post about it:

Using time-based indices is typically better because removing entire
indices is super-fast (really remove the associated files and little
metadata) and doesn't mess up with the rest of your data and caches.

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Tue, Apr 29, 2014 at 4:04 PM, Abrar Sheikh abrar2002as@gmail.com wrote:

Hi,

I would what to know how to rotate old records from ES index, coz when i
delete documents with delete query API it does not free up disk space. Eg.
i want to remove records older than 3 days in my index. For me disk space
is an issue.

Thanks and Regards,
Abrar.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/61d35912-72ad-4139-aec3-46fc89f36ac9%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/61d35912-72ad-4139-aec3-46fc89f36ac9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3VZzmoJ5J8GSh88J%2BXN%2BOGyPh-bwg7W5r6QokKOMsdoQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

1 Like