I have installed Curator and ran a delete command as follows:
curator --config /opt/software/elk/elasticsearch-6.2.4/curator/curator.yml /opt/software/elk/elasticsearch-6.2.4/curator/action_delete.yml
The delete worked with no issues and cleaned up exactly what was expected. However after the delete I started getting the below error message in elasticsearch log:
[2018-07-25T16:15:42,308][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:140) ~[?:?]
This caused no records to pass elasticsearch and stopped my dashboards. I had to delete all the indicies to stop the error message and get indicies to continue to gather data. Any ideas on why this happened or how to prevent would be greatly appreciated.
The curator.yml is as follows:
client:
hosts:
- aleas031.vwoa.na.vwg
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth: elastic:xxxxxxx
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile: /data/apps/elk/elasticsearch/logs/curator.log
logformat: default
blacklist: []
action_delete.yml:
actions:
1:
action: delete_indices
description: >-
Delete indices older than 5 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: ipstash-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 7
2:
action: delete_indices
description: >-
Delete indices older than 5 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: wasstash-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 7
3:
action: delete_indices
description: >-
Delete indices older than 5 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
options:
ignore_empty_list: True
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: metricbeat-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 2