Delete old data from an Index

Hello,
How can i delete the old data from an index, if i create an index everyday i am facing some problems wrt shards performance , so i have only one index and i would like to delete old data. please let me know how can i achieve this

You need to use the delete by query plugin, but it is really inefficient.

Use weekly/monthly and reduce the shard count. That'll be much cheaper than the above.

Using time-based indices does not necessarily mean daily indices. If you have long retention period and reasonably small amounts of data coming, you can instead use monthly indices. With a single index you will need to use the delete by query API, which will be more resource intensive than simply dropping an index.

is there a way to automate this?

If you are using time-based indices, you can use Curator to automate index management.

1 Like

i dont prefer using time based indices, i would like to have single index , can you please throw some light on API for clearing the data till last week?

Using a range query based on your timestamp field together with the delete-by-query API should work.

You should really follow @Christian_Dahlqvist and @warkolm advices. Time based indices is the way to go. You will also benefit from index sorting, so faster searches.

any references for deleting old data with powershell which is n days old?:slight_smile:

If you are using time based indices, that should be something like:

curl -XDELETE http://localhost:9200/test-2017-06

(let say you have monthly indices).

1 Like

The aforementioned Curator comes with a Windows installer, so you'd be free to use it in power-shell.

I should also point out it has command-line functionality as well (not just YAML-configuration), which should be just as easy to use from powershell.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.