Dangling Indices in Ingest nodes after deleting Indices with Currator

Elastic Stack Elasticsearch
1

Hi,

After running curator for deleting indicies older than two weeks, I see following error in Ingest nodes elasticsearch logs!

tailf /var/log/elasticsearch/dbus.log 

[2018-02-19T10:26:09,264][WARN ][o.e.g.DanglingIndicesState] [ip-10-50-30-150] [[logstash-demo-all-container-2018.06/LQF1f6_WQ3CMDO2qYXIa6Q]] can not be imported as a dangling index, as an index with the same name and UUID exist in the index tombstones.  This situation is likely caused by copying over the data directory for an index that was previously deleted.
[2018-02-19T10:26:09,264][WARN ][o.e.g.DanglingIndicesState] [ip-10-50-30-150] [[logstash-demo-all-application-2018.06/iuSMlXguQZODV3jN5I2tfw]] can not be imported as a dangling index, as an index with the same name and UUID exist in the index tombstones.  This situation is likely caused by copying over the data directory for an index that was previously deleted.
[2018-02-19T10:26:09,264][WARN ][o.e.g.DanglingIndicesState] [ip-10-50-30-150] [[logstash-demo-all-script-2018.06/uB5SavnPSbe_JIBkUoUrjQ]] can not be imported as a dangling index, as an index with the same name and UUID exist in the index tombstones.  This situation is likely caused by copying over the data directory for an index that was previously deleted.
[2018-02-19T10:26:09,264][WARN ][o.e.g.DanglingIndicesState] [ip-10-50-30-150] [[logstash-demo-all-cron-2018.06/Om6NSFQATseZvk28EhHrfw]] can not be imported as a dangling index, as an index with the same name and UUID exist in the index tombstones.  This situation is likely caused by copying over the data directory for an index that was previously deleted.
[2018-02-19T10:26:09,264][WARN ][o.e.g.DanglingIndicesState] [ip-10-50-30-150] [[logstash-demo-all-haproxy-2018.06/CrksY5NCTIGpbBFzFKksPw]] can not be imported as a dangling index, as an index with the same name and UUID exist in the index tombstones.  This situation is likely caused by copying over the data directory for an index that was previously deleted.
[2018-02-19T10:26:09,264][WARN ][o.e.g.DanglingIndicesState] [ip-10-50-30-150] [[logstash-demo-all-access-2018.06/ZKN-rpgKQu6spuppLDeufg]] can not be imported as a dangling index, as an index with the same name and UUID exist in the index tombstones.  This situation is likely caused by copying over the data directory for an index that was previously deleted.

My elasticsearch cluster are consists of eight nodes that are as following:

  • Three Master nodes (10.50.45.124,10.50.40.180, 10.50.30.106)
  • Two Ingest nodes (Running Kibana and Curator)
    *Three Data nodes

Curator is installed on the both ingest nodes but it is pointed to run the action file (DELETING OLD INDICES) on the master nodes as you can see in below I put IPs for master nodes in curator config file.

cat /opt/elasticsearch-curator/curator.yml 
---
client:
  hosts:
    - 10.50.45.124
    - 10.50.40.180
    - 10.50.30.106
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False
logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Curator action files is as following:

cat /opt/elasticsearch-curator/delete_indices.yml 
---
actions:
  1:
    action: delete_indices
    description: Delete indices older than one week for "logstash-playground-all-" prefixed indices.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-playground-all-*-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%W'
      unit: weeks
      unit_count: 1
      exclude:

  2:
    action: delete_indices
    description: Delete indices older than two weeks for "logstash-demo-all-" prefixed indices.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-demo-all-*-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%W'
      unit: weeks
      unit_count: 2
      exclude:

  3:
    action: delete_indices
    description: Delete indices older than one month for "logstash-production-all-" prefixed indices.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-production-all-*-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: months
      unit_count: 1
      exclude:

  4:
    action: delete_indices
    description: Delete indices older than two months for "logstash-production-kafka-" prefixed indices.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-production-kafka-*-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: months
      unit_count: 2
      exclude:

I can remove these dangling indices from /var/lib/elasticsearch/nodes/0/ with a bash script but as it is risky I want to find a way to solve this issue with the curator!

I am looking forward for your suggestion and help.
Thank you very much

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.