Alternative for Delete By Query or Solution for Performance improvement

Elastic Stack Elasticsearch
1

Hi ,

We have index in ELK ,where we are storing transaction details ,where we want to delete some records after some time.

As quantity of requests is so high ,its taking so much time to complete the execution of the Delete query(almost 4-5 hours) .

So is there alternative to delete this data without any performance issue or any alternative to Delete by Query which will make deletion faster.

Please suggest.

Thanks ,
Mrunalini

2

Deleting docs is expensive but deleting whole indices is quick/cheap.
This is why we advocate using "time-based indices" e.g. keeping one per month and then moving old indices to backup storage or dropping whole indices after a time. This practice has been formalised in the idea of datastreams

1 Like
3

We already have monthly indexes , but we cant afford to loose whole data ,so is there any possible way we can increase Delete Query performance .

4

Use weekly indices?
Or are you saying not all documents in a time-based index expire at the same time?

5

Hi Mark,

Thank you for instant reply !!

We have Index where we are logging documents with request and response payload for few number of transaction ,not for all the transaction.

So what we are thinking is we can delete the just request -response payload of that particular transaction after 15 days may be, so that storage issue will not be there.

So we have problem for using DBQ for this as it take so much time.
Please suggest.

Thanks and regards,
Mrunalini

6

Keep 15 indexes each with a day’s worth of content. Each day create a new one and delete the oldest - this is a typical time-based indexing policy.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.