How can curator delete empty indices?

Elastic Stack Elasticsearch
1

Hi,
We are using rollover with curator with max_age + max_size. In our automated index creation process we create multiple indices (think environments like like, dev, test, pre-prod etc ) BUT all of these indices are not always used so there can be 0 documents in them.

When an index is rolled over (creation_date > max_age) our index is not removed since we use field_stats filter and remove indices by timestamp.

I have not found a way to remove empty indices, can anyone help we out with this?

Regards /Johan

2

Field stats can’t do a document count on an empty index. You’ll have to figure out a different attribute to identity your indices. creation_date is still available, and index name.

You may also have to use your own scripting with the _cat API to identify empty indices and delete them with curl, or a similar tool.

3

Thanks @theuntergeek,

I only want to check if doc_count is 0, nothing else. I already exclude all indices with a write alias. Is it still not possible?

I could easily fix this with ansible and the http api but it would be nice if curator could fix it for me.

Regards /Johan

4

There is no doc count filter, so you'd have to do some other clever work with filters to use other metrics if you wanted to use Curator. Either that, or script up some python using the Curator API.

5

Hi,

Ok, I understand. Is a doc count filter something that would be of interest for curator if I try to add this feature?
Regards /Johan

6

@theuntergeek,

I can see from the code that there already is a filter_empty function but it is not exposed as a filter (https://github.com/elastic/curator/blob/5fdcc337625c6a7e9aeeb90de625b54b92cbbcb7/curator/indexlist.py#L722).
A simple (not sure if it is the best though) would be to expose this as a new filter

- filtertype: empty
exclude: False

Or is there any reason why this would not fit in curator?

Regards /Johan

7

Oh, yeah. I did make that. It's used on line 301. I haven't exposed it for general use, but I suppose since it's there, it could be.

It would need to be added to:

  • Documentation
  • The filter schema(s)
  • ???

There are already tests for it, so I think that might be all that's required to add it.

8

Great! I am on it, I will send and PR.

/Johan

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.