Issues with indicies after running Curator delete

I have installed Curator and ran a delete command as follows:

curator --config /opt/software/elk/elasticsearch-6.2.4/curator/curator.yml /opt/software/elk/elasticsearch-6.2.4/curator/action_delete.yml

The delete worked with no issues and cleaned up exactly what was expected. However after the delete I started getting the below error message in elasticsearch log:

[2018-07-25T16:15:42,308][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:140) ~[?:?]

This caused no records to pass elasticsearch and stopped my dashboards. I had to delete all the indicies to stop the error message and get indicies to continue to gather data. Any ideas on why this happened or how to prevent would be greatly appreciated.

The curator.yml is as follows:
client:
hosts:
- aleas031.vwoa.na.vwg
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth: elastic:xxxxxxx
timeout: 30
master_only: False

logging:
loglevel: INFO
logfile: /data/apps/elk/elasticsearch/logs/curator.log
logformat: default
blacklist: []

action_delete.yml:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 5 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: ipstash-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 7
2:
action: delete_indices
description: >-
Delete indices older than 5 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: wasstash-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 7
3:
action: delete_indices
description: >-
Delete indices older than 5 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: metricbeat-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 2

@rfelske

Hope i can help but i need some information.

  1. Which environment you were using
  2. And which version of python you were using.

Thanks

I am using python 2.7.5, I'm not sure what you mean by environment. I am using a basic license.
FYI

Curator did not cause this. It did what you configured it to do. This is something upstream:

Your Curator isn’t configured to manage the monitoring indices, -monitoring-. You might want to look at those indices, and whether any monitoring changes were made, or added.

Thanks for the info. I am still unsuer why this occurred but as this was a POC I have done a complete re-install for production use and curator is working fine there.
Thanks,

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.