Curator forcemerge Exception

[Daniel314 (Daniel C)]

Hi,

I have an Elastic cluster running version 8.12.x with curator_cli version 8.0.10.
I recently had an odd timeout/connect issue with curator_cli while it was doing a forcemerge (scheduled script -- not the focus of this post), so when I noticed the issue I manually re-ran the command (forcemerge), and it errored out with a very generic exception.

After some digging, it appears to be a variant of this issue (curator_cli throws an error because the specified index segments are already at/below the specified number of segments, even though --ignore_empty_list has been specified). However, now the error seems even more obfuscated (plus the script exits with a non-zero exit code). When running in debug mode, here are the last two lines of output:

2024-02-27 07:37:18,632 DEBUG          curator.indexlist       empty_list_check:400  Checking for empty list
2024-02-27 07:37:18,632 CRITICAL  curator.cli_singletons.cli_action.forcemerge    do_singleton_action:224  Failed to complete action: forcemerge.  <class 'Exception'>:

Ideally, curator_cli forcemerge would silently ignore any index at/below the specified number of segments. However, if the curator_cli error isn't going to be suppressed, could the error message at least be more specific about what the error is?

Thanks,

 - Daniel

[theuntergeek (Aaron Mildenstein)]

This should be a feature request at GitHub, if that's something worth pursuing.

One of the other issues that this may be is that force merges once started cannot be canceled. If you immediately restarted Curator and the force merge in question was still ongoing, it is possible that the error is related to this phenomenon. I'm not in front of the code base at the moment, but I believe it would be trivial to check for the expected number of segments and send an INFO log level that it is already at that level, if it doesn't already do this.

[Daniel314 (Daniel C)]

The 8.0.10 version of curator_cli doesn't output an INFO-level message about already being at/below the requested number segments. I'll add a feature request on the GitHub site -- thanks.

In my case, there wasn't any other forcemerge going on. If I pick on other indices from prior forcemerges that are already at one segment per shard, I receive the same generic exception error. Here's an example of that:

$ curator_cli --version
curator_cli, version 8.0.10
$ curl -sS -XGET http://localhost:9200/myindex-2024.02.23/_segments | jq '[ .. | .shards? // empty | to_entries[] | { "key":"shard \(.key)", "value":.value[0].num_committed_segments }] | flatten | from_entries'
{
  "shard 0": 1,
  "shard 1": 1
}
$ curator_cli --loglevel INFO  --request_timeout 14400 forcemerge --ignore_empty_list --max_num_segments 1 --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"myindex-2024.02.23"}]'
Using default configuration file at /.../.curator/curator.yml
2024-02-27 09:43:17,889 CRITICAL  Failed to complete action: forcemerge.  <class 'Exception'>: