ConflictError(409, ' running curator

Elastic Stack Elasticsearch
1

Yesterday I received help here that allowed me to get curator to run. I tested my action.yml in dry-run mode yesterday - and all looked fine.

I am brand new to ELK and Curator - so please bear with me

This morning - I went to run the action for real - and got an error.

Here is my action.yml:

# cat /etc/curator/action.yml
actions:

  • 1:*

  • action: close*

  • description: >-*

  •  Close indices older than 30 days (based on creation date).*

  • options:*

  •  ignore_empty_list: True*

  •  delete_aliases: False*

  •  disable_action: False*

  • filters:*

    • filtertype: age*
  •  source: creation_date*

  •  direction: older*

  •  timestring: '%Y.%m.%d'*

  •  unit: days*

  •  unit_count: 30*

  • 2:*

  • action: delete_indices*

  • description: >-*

  •  Delete indices older than 30 days (based on creation date).*

  • options:*

  •  ignore_empty_list: True*

  •  disable_action: False*

  • filters:*

    • filtertype: age*
  •  source: creation_date*

  •  direction: older*

  •  timestring: '%Y.%m.%d'*

  •  unit: days*

  •  unit_count: 30*

When I run in dry-run mode - I get job completed. When I run 'for real' I pulled this error from the log (debug). This is the last two lines of the log #724 and #725:

*724 2020-03-26 12:18:48,053 INFO       curator.actions.close              do_action:305  Closing 15 selected indices: ['aapc.log-stage', 'aapc.log-prod', 'aapc.log-local', '.kibana6_        8', '.kibana6_7', 'searchguard', '.kibana_task_manager', 'shipapidev-', 'wwwdev-', 'aapc.log-test', '.tasks', 'aapc.log-navigators', '.kibana6_9', '.security', '.kibana']*
  • 725 2020-03-26 12:18:48,981 ERROR curator.cli run:213 Failed to complete action: close. <class 'curator.exceptions.FailedExecution'>: Exception en countered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConflictError(409, '{"_shards":{"total":79,"successful":41,"failed":2},"aap c.log-prod":{"total":10,"successful":3,"failed":2,"failures":[{"shard":0,"reason":"pending operations","routing":{"state":"STARTED","primary":true,"node":"hNGMw-wTSZyXMuMJoJG4QQ" ,"relocating_node":null,"shard":0,"index":"aapc.log-prod","allocation_id":{"id":"IWU3OJVsTPq4rh04uvZCQg"}}},{"shard":1,"reason":"pending operations","routing":{"state":"STARTED", "primary":true,"node":"hNGMw-wTSZyXMuMJoJG4QQ","relocating_node":null,"shard":1,"index":"aapc.log-prod","allocation_id":{"id":"NzFW9d9tTPCF6CsJuU3bwQ"}}}]},".kibana_task_manager" :{"total":1,"successful":1,"failed":0},"aapc.log-test":{"total":10,"successful":5,"failed":0},"shipapidev-":{"total":10,"successful":5,"failed":0},".tasks":{"total":1,"successful ":1,"failed":0},"aapc.log-navigators":{"total":10,"successful":5,"failed":0},".security":{"total":1,"successful":1,"failed":0},"searchguard":{"total":1,"successful":1,"failed":0} ,".kibana":{"total":2,"successful":1,"failed":0},"aapc.log-stage":{"total":10,"successful":5,"failed":0},".kibana6_9":{"total":1,"successful":1,"failed":0},"aapc.log-local":{"tot al":10,"successful":5,"failed":0},".kibana6_8":{"total":1,"successful":1,"failed":0},".kibana6_7":{"total":1,"successful":1,"failed":0},"wwwdev-":{"total":10,"successful":5,"fail ed":0}}')*

If I read this right, the error is during the close section of action.yml. It never gets to the delete at all.

2

So this 409 error is indicating that you're trying to close aapc.log-prod, which could not perform a synced flush because of pending operations, which suggests that new data is still trying to be indexed/added.

Are you sure you want to close an index which is actively receiving data?

3

Probably not - but the operation wouldn't try to close it unless it was more than 30 days old. Correct? Is this typical? How can I push the pending operations through? or exclude aapc.log-prod somehow?

4

If there are pending operations and I don't close the indices before I run the delete-indices action, what happens?

5

There is no need to close an index before deleting it. You can delete without closing.

6

this took care of my issue. Thank You!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.