TTL Documents

Elastic Stack Elasticsearch
1

Per TTL Documents, Shield and Found | Elastic:

Note that in the current version of Elasticsearch, _ttl is deprecated.

What is the approach for modern elasticsearch 6.5+ (in terms of documents that needs to be expired)?

Please advise.

2

Do these documents have the same expiration period or it vary widely?

3

each doc to have own expiration, but they all would have same ttl

4

In this case the cheapest solution would be to create a new index periodically. For example, if you have 6 months ttl, you can create a new index every month. While searching you can add a filter that will filter out documents that are older than 6 months. After 6 month, you will create a new index and delete completely the oldest index. You can automate this operation using curator.

5

What about if ttl of document is like 7 days (not 6 month)? also isn't curator for 5.x?

6

Hi Alexus,

Curator 5.x can be used with elasticsearch 6.x, as can be seen in this Curator Version Compatibility Matrix.

Furthermore, in this scenario, you might want to combine Igor's solution and the use of the rollover api. You can take a look at this this blog post to learn more :wink:

Best regards,

Charles Casadei

7

Just create and a new index and delete the week-old index every day.

8

... not ideal solution like ttl per doc, but I guess that could work too)

Thank you!

9

That's the most optimal solution. You can obviously periodically run delete_by_query request, but this is going to be very inefficient since you will be deleting the oldest records that will end up in biggest segments by the time you will be deleting them, which means your biggest segments will have a lot of holes in them, requiring merging, wasting space and CPU and slowing things down. This is pretty much the worst-case scenario performance wise.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.