How to find rolled indices Eligible to Shrink programmatically

Elastic Stack Elasticsearch
1

Trying to use curator to shrink any indices that have been rolled nightly. Any way to use the filtering in curator to identify a rolled index? I believe is_write_index is set to false but I'm having trouble finding that via Curator filters or API.

2

That's a relatively new thing, and hasn't been added to Curator yet. Not all versions of Elasticsearch support that flag. Certainly not all the ones that the current Curator release supports.

At any rate, it's been added as an issue already: https://github.com/elastic/curator/issues/1358

3

That makes sense. How else have you seen people identify their rolled indices for shrinking via Curator?

4

First use the pattern filter to select only indices matching your rollover index pattern. Then, use the alias filter with the exclude option to select indices not associated with the rollover alias.

5

For example:

actions:
  1:
    action: shrink
    description: Shrink rolled indices
    options:
      # ...
    filters:
    - filtertype: pattern
      kind: prefix
      value: indexpattern-
    - filtertype: alias
      aliases: myrolloveralias
      exclude: true
6

EDIT:
Nevermind, I think the right approach is to first execute a shrink process and if I want to execute a re-allocation of any indices that have been shrunk I can utilize a allocation job. Thanks again, works like a charm.

ORIGINAL:
This worked great, thanks!! I think I am having some confusion though with the shrink_node: field. My setup consists of 2 nodes for testing, 1 box_type hot, 1 box_type cold. The idea is all writes will occur on the hot based on index template routing and the shrink target will always go to the cold. However, I am finding that if I set shrink_node to DETERMINISTIC that in the log output 2019-02-04 20:47:03,138 INFO Moving shards to shrink node: "hot-box" yet, the new index will be created on the "cold-box". This is great, as my goal is for all shrinks to be on the cold box, but I find that once the amount of data on the cold box exceeds the hot-box, the shrink indices will be created on the hot-box. The problem is when I set shrink_node explicitly to "cold-box" it fails with the error that Failed to complete action: shrink. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: Unable to shrink index "test-tooling-000004" as not all shards were found on the designated shrink node (cold-box): .

This just seems really odd, because when DETERMINISTIC is set and the data is lower the cold-box is able to allow for the shrink to occur and land the index on it, but when set explicitly this exception is thrown :(. Am I misunderstanding the purpose of the shrink_node value?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.