Hi Team,
I am trying to use curator to do some housekeeping and trying to select only the indexes that have not been housekept before.
On a high level, I am doing this: close some old indexes => set index setting index codec to best compression => re open those indexes => then force merge. This all works great, no issues.
Now a small problem comes when I try to do something a bit more clever where I am trying to select indexes older than certain age and at the same time the indexes should not be already force merged: Although between actions I use the same set of filters, these filters do not return the same list of indexes depending on the current state of the indexes.
Let me explain using the pseudo curator actions config below:
actions:
1:
action: close
options: # some opts
...
filters:
# use age and forcemerged filters to select only
# indexes older than some threshold and exclude forcemerged indexes
# here both dry-run and real run return desired list of indexes
2:
action: index_settings
options: # some opts
...
filters:
# same filters as above, use age and forcemerged filters to select only
# indexes older than some threshold and exclude forcemerged indexes
# here though, dry-run shows the correct desired list of indexes
# but real run does not behave as expected, because by the looks of it
# because indexes are closed the filtering fails at checking whether
# indexes had already been forcemerged or not hence resulting in
# curator.exceptions.NoIndices exception
3:
# subsequent actions also fail due to the same reason
Besides that certainly dry-run should show expected run behaviour, now what I mean from the topic subject by "index list reuse" is currently the filters are per action, I was wondering (although I am pretty sure it's not possible for the time being) whether the filters could be made also a few levels up to apply across all actions or if there is some other way to achieve index list re-use, eg:
actions:
1:
action: first action
2:
action: another one
3:
# etc
filters:
# curator gets a list of indexes once then the same list is passed
# as an argument to all actions
This I think should also help make the dry-run more consistent.
Many thanks for your advice,
P.S. As this is most likely not possible at the moment, my workaround option is to use filters to implement a sliding window without filtering by forcemerged: age older than X and younger than (X + say 1 week) and run curator only every week - this ensures I don't do housekeeping against the ever growing list of older indexes over again and again.