Curator_cli, indices, and a new server to me

[bithead008 (Chris M)]

I turned on debug output because the indices don't seem to be getting cleaned and the machine is running out of disk space. This machine was restored from a backup and I believe that it wasn't quiesced so perhaps Elasticsearch is unhappy about some corruption?

I have no idea how to remedy this as this is a new server to me. Any hints you guys can give me?

curator_cli --version

curator_cli, version 5.6.0

/usr/local/sbin/clean_indices.sh

• curator_cli --loglevel DEBUG --logfile /tmp/clean_indices.log --host 10.155.6.20 --port 8881 --ssl-no-validate --http_auth logreader:SomePassword --timeout=180 delete_indices --filter_list '{"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":60,"direction":"older"}'

more /tmp/clean_indices.log

2019-03-27 13:57:55,212 DEBUG curator.cli_singletons.cli_action.delete_indices check_options:96 Validating provided options: {'allow_ilm_indices': False}
2019-03-27 13:57:55,215 DEBUG curator.validators.SchemaCheck init:26 Schema: {'allow_ilm_indices': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function
Boolean at 0x7f3444eb26a8>, msg=None), msg=None), 'continue_if_exception': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f3444eb2840>, msg=None), msg=Non
e), 'disable_action': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f3444eb29d8>, msg=None), msg=None), 'ignore_empty_list': Any(<class 'bool'>, All(Any(<
class 'str'>, msg=None), <function Boolean at 0x7f3444eb2b70>, msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None)}
2019-03-27 13:57:55,216 DEBUG curator.validators.SchemaCheck init:27 "options" config: {'allow_ilm_indices': False}
2019-03-27 13:57:55,216 DEBUG curator.cli_singletons.cli_action.delete_indices check_filters:117 Validating provided filters: [{'filtertype': 'age', 'source': 'name', 'time
string': '%Y.%m.%d', 'unit': 'days', 'unit_count': 60, 'direction': 'older'}]
2019-03-27 13:57:55,216 DEBUG curator.validators.SchemaCheck init:26 Schema: <function Filters..f at 0x7f3444eb2378>
2019-03-27 13:57:55,216 DEBUG curator.validators.SchemaCheck init:27 "filters" config: [{'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit':
'days', 'unit_count': 60, 'direction': 'older'}]
2019-03-27 13:57:55,216 DEBUG curator.defaults.filtertypes age:55 AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minute
s', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(<class 'str'>, msg=None)}, {'epoch': Any(Coerce(int, msg=No
ne), None, msg=None)}, {'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f3444f1c488>, msg=None), msg=None)}, {'source': Any('name', 'creation_dat
e', 'field_stats', msg=None)}, {'stats_result': Any('min_value', 'max_value', msg=None)}, {'timestring': Any(<class 'str'>, msg=None)}]
2019-03-27 13:57:55,217 DEBUG curator.validators.SchemaCheck init:26 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forc
emerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', '
forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'ho
urs', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'epoch': Any(Coerce(int, msg=None), None, ms
g=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f3444f1c488>, msg=None), msg=None), 'source': Any('name', 'creation_date', 'field_stats'
, msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'timestring': Any(<class 'str'>, msg=None)}
2019-03-27 13:57:55,217 DEBUG curator.validators.SchemaCheck init:27 "filter" config: {'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit': 'd
ays', 'unit_count': 60, 'direction': 'older'}
2019-03-27 13:57:55,217 DEBUG curator.validators.filters f:48 Filter #0: {'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit': 'days', 'uni
t_count': 60, 'direction': 'older', 'epoch': None, 'stats_result': 'min_value', 'exclude': False}
2019-03-27 13:57:55,217 DEBUG curator.utils get_client:802 kwargs = {'hosts': ['10.155.6.20'], 'port': 8881, 'use_ssl': False, 'timeout': 180, 'master_only': Fa
lse, 'url_prefix': '', 'aws_sign_request': False, 'http_auth': 'logreader:SomePassword', 'ssl_no_validate': True, 'certificate': None, 'aws_secret_key': None, 'aws_token': None, 'client_cert
': None, 'aws_key': None, 'client_key': None}
2019-03-27 13:57:55,221 DEBUG curator.utils get_client:879 Not using "requests_aws4auth" python module to connect.

[A_B]

When troubleshooting curator_cli I always start as simple as possible and try to just list some indices. Once that works then I try to action them...

I have a wrapper script for usr/local/bin/curator_cli which is

# cat /usr/local/bin/curator_delete.sh
#!/bin/bash

set -o nounset
set -o errexit

usage="$(basename "$0") [-h] [-p foo] [-s bar] [-d n] -- delete old Elasticsearch indices using curator

where:
    -h  show this help text
    -p  index pattern (start of index name)
    -s  Elasticsearch server to run command against
    -d  delete indices older than, in days (default: )
    -x  add to -d to delete more indices (for use with sensu check hooks)"

DAYS=
EXTRA_DAYS=0

while getopts ":hp:s:d:x:" option; do
  case $option in
    h) echo "$usage"
       exit
       ;;
    d) DAYS=$OPTARG
       ;;
    x) EXTRA_DAYS=$OPTARG
       ;;
    p) PATTERN=$OPTARG
       ;;
    s) SERVER=$OPTARG
       ;;
  esac
done
shift $((OPTIND -1))

(( DAYS_TOTAL = $DAYS - $EXTRA_DAYS ))

generate_post_data()
{
  cat <<EOF
{
  "filtertype":"age",
  "source":"creation_date",
  "direction":"older",
  "unit":"days",
  "unit_count":"$DAYS_TOTAL"
},
{
  "filtertype":"pattern",
  "kind":"prefix",
  "value":"$PATTERN"
}
EOF
}

/usr/local/bin/curator_cli \
  --loglevel INFO \
  --logfile /var/log/curator.log \
  --host $SERVER \
  delete_indices \
  --filter_list "[$(generate_post_data)]"

Hope that helps. I do not need authentication or anything like that so it is pretty straight forward...