Curator - Filters output inconsistency - Depending on action type : delete or shrink

ORich · March 20, 2019, 4:01pm

I run curator (5.6) from command line:

curator /opt/elasticsearch-curator/shrink_metricbeat_indices.yml --config /opt/elasticsearch-curator/curator.config.yml --dry-run

No indice match as per output:

2019-03-20 16:46:23,208 INFO      Preparing Action ID: 1, "shrink"
2019-03-20 16:46:23,214 INFO      Trying Action ID: 1, "shrink":
2019-03-20 16:46:23,296 INFO      Skipping action "shrink" due to empty list: <class 'curator.exceptions.NoIndices'>
2019-03-20 16:46:23,296 INFO      Action ID: 1, "shrink" completed.
2019-03-20 16:46:23,296 INFO      Job completed.

Then I run:

curator /opt/elasticsearch-curator/delete_metricbeat_indices.yml --config /opt/elasticsearch-curator/curator.config.yml --dry-run

Two indices are matching as per output:

2019-03-20 16:56:57,662 INFO      Preparing Action ID: 1, "delete_indices"
2019-03-20 16:56:57,668 INFO      Trying Action ID: 1, "delete_indices"
2019-03-20 16:56:57,756 INFO      DRY-RUN MODE.  No changes will be made.
2019-03-20 16:56:57,756 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2019-03-20 16:56:57,756 INFO      DRY-RUN: delete_indices: metricbeat-6.5.1-2019.03.19 with arguments: {}
2019-03-20 16:56:57,756 INFO      DRY-RUN: delete_indices: metricbeat-6.5.1-2019.03.20 with arguments: {}
2019-03-20 16:56:57,756 INFO      Action ID: 1, "delete_indices" completed.
2019-03-20 16:56:57,756 INFO      Job completed.

Both shrink_metricbeat_indices.yml and delete_metricbeat_indices.yml have same filter section:

    filters:
    - filtertype: pattern
      kind: regex
      value: '^(metricbeat-).*$'
      exclude:

theuntergeek · March 20, 2019, 4:19pm

First, please encapsulate pasted log and config data between two sets of triple back-ticks, like this:

```
PASTE HERE
```

This improves readability for anyone trying to see what you have pasted.

Second, please run this again with debug level logging. It shows why filters behave the way they do, and what decisions it makes.

Technically speaking, the filter code is uniform. There is not separate or different filter code for any of the actions. However, shrink probably does some other things behind the scenes (I don't recall offhand, and I'm not able to look right now).

ORich · March 20, 2019, 5:08pm

Thanks for your swift reply.
According to here enclosed log file snapshot, metricbeat indices are filtered out by the number of shards. But still don't understand why ? Here is the yaml file.
Thx for support. Regards. Richard

---
actions:
  1:
    action: shrink
    description: >-
    options:
      ignore_empty_list: True
      shrink_node: DETERMINISTIC
      shrink_suffix: '-shrink'
      node_filters:
        permit_masters: True
      number_of_shards: 1
      number_of_replicas: 0
      wait_for_active_shards: 1
      extra_settings:
        settings:
          index.codec: best_compression
      post_allocation:
        key: node_type
        value: cold
      delete_after: True
      wait_for_completion: True
      continue_if_exception: True
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(metricbeat-).*$'
      exclude:

2019-03-20 18:00:39,708 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2019-03-20 18:00:39,708 DEBUG          curator.indexlist          filter_closed:722  Index metricbeat-6.5.1-2019.03.19 state: open
2019-03-20 18:00:39,708 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-6.5.1-2019.03.19 is actionable and remains in the list.
2019-03-20 18:00:39,708 DEBUG          curator.indexlist          filter_closed:722  Index metricbeat-6.5.1-2019.03.20 state: open
2019-03-20 18:00:39,708 DEBUG          curator.indexlist           __actionable:35   Index metricbeat-6.5.1-2019.03.20 is actionable and remains in the list.
2019-03-20 18:00:39,708 DEBUG          curator.indexlist       filter_by_shards:1013 Filtering indices by number of shards
2019-03-20 18:00:39,709 DEBUG          curator.indexlist       empty_list_check:226  Checking for empty list
2019-03-20 18:00:39,709 DEBUG          curator.indexlist           working_list:237  Generating working list of indices
2019-03-20 18:00:39,709 DEBUG          curator.indexlist       filter_by_shards:1031 Filter by number of shards: Index: metricbeat-6.5.1-2019.03.19
2019-03-20 18:00:39,709 DEBUG          curator.indexlist       __not_actionable:39   Index metricbeat-6.5.1-2019.03.19 is not actionable, removing from list.
2019-03-20 18:00:39,709 DEBUG          curator.indexlist       filter_by_shards:1031 Filter by number of shards: Index: metricbeat-6.5.1-2019.03.20
2019-03-20 18:00:39,709 DEBUG          curator.indexlist       __not_actionable:39   Index metricbeat-6.5.1-2019.03.20 is not actionable, removing from list.

ORich · March 20, 2019, 5:18pm

Here is the health status for one of the metricbeat indice

[root@i-00213c2c-elk-server-15486830851 centos]# curl -X GET "localhost:9200/_cluster/health/metricbeat-6.5.1-2019.03.20?pretty"
{
  "cluster_name" : "MyClusterES",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

theuntergeek · March 20, 2019, 5:49pm

How can you shrink an index that already only has one primary shard? The shrink api is for reducing shard counts, so this index would be ineligible. That’s why Curator omitted it.

Update: Expanded answer.

You specified:

…in your YAML configuration. The shrink action in Curator reads this, and omits indices which already have that number of shards. This is why you experienced a different outcome between the two actions, even though they had identical filters. Curator is removing unnecessary actions for you (shrinking an index which is already at 1 primary shard).

ORich · March 21, 2019, 9:01am

Many thanks Aaron.
Your last answer is crystal clear. Since metricbeat indice size is huge - roughly 40 Gb - I had in mind to benefit from "shrink index.codec: best_compression" feature to reduce disk space usage ... even if metricbeat is mapped to a single primary shard.
Does it make sense to setup 2 primary shards per metricbeat indices in order to apply shrink to one primary shard with compression and then have a less disk usage at the end ?
Regards.

theuntergeek · March 22, 2019, 8:30pm

If 2 primary shards are unnecessary for your use case, then doing that just to achieve best compression is a bad use case for the shrink action in Curator.

You can achieve the same thing with a series of actions:

Close selected indices.
Change settings on the same indices.
Re-open the same indices.
Perform a force merge on those indices (this will trigger the actual compression of data).

system · April 19, 2019, 8:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.