ConflictError(409, ' running curator

Yesterday I received help here that allowed me to get curator to run. I tested my action.yml in dry-run mode yesterday - and all looked fine.

I am brand new to ELK and Curator - so please bear with me

This morning - I went to run the action for real - and got an error.

Here is my action.yml:

# cat /etc/curator/action.yml
actions:

  • 1:*

  • action: close*

  • description: >-*

  •  Close indices older than 30 days (based on creation date).*
    
  • options:*

  •  ignore_empty_list: True*
    
  •  delete_aliases: False*
    
  •  disable_action: False*
    
  • filters:*

    • filtertype: age*
  •  source: creation_date*
    
  •  direction: older*
    
  •  timestring: '%Y.%m.%d'*
    
  •  unit: days*
    
  •  unit_count: 30*
    
  • 2:*

  • action: delete_indices*

  • description: >-*

  •  Delete indices older than 30 days (based on creation date).*
    
  • options:*

  •  ignore_empty_list: True*
    
  •  disable_action: False*
    
  • filters:*

    • filtertype: age*
  •  source: creation_date*
    
  •  direction: older*
    
  •  timestring: '%Y.%m.%d'*
    
  •  unit: days*
    
  •  unit_count: 30*
    

When I run in dry-run mode - I get job completed. When I run 'for real' I pulled this error from the log (debug). This is the last two lines of the log #724 and #725:

*724 2020-03-26 12:18:48,053 INFO       curator.actions.close              do_action:305  Closing 15 selected indices: ['aapc.log-stage', 'aapc.log-prod', 'aapc.log-local', '.kibana6_        8', '.kibana6_7', 'searchguard', '.kibana_task_manager', 'shipapidev-', 'wwwdev-', 'aapc.log-test', '.tasks', 'aapc.log-navigators', '.kibana6_9', '.security', '.kibana']*
  • 725 2020-03-26 12:18:48,981 ERROR curator.cli run:213 Failed to complete action: close. <class 'curator.exceptions.FailedExecution'>: Exception en countered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConflictError(409, '{"_shards":{"total":79,"successful":41,"failed":2},"aap c.log-prod":{"total":10,"successful":3,"failed":2,"failures":[{"shard":0,"reason":"pending operations","routing":{"state":"STARTED","primary":true,"node":"hNGMw-wTSZyXMuMJoJG4QQ" ,"relocating_node":null,"shard":0,"index":"aapc.log-prod","allocation_id":{"id":"IWU3OJVsTPq4rh04uvZCQg"}}},{"shard":1,"reason":"pending operations","routing":{"state":"STARTED", "primary":true,"node":"hNGMw-wTSZyXMuMJoJG4QQ","relocating_node":null,"shard":1,"index":"aapc.log-prod","allocation_id":{"id":"NzFW9d9tTPCF6CsJuU3bwQ"}}}]},".kibana_task_manager" :{"total":1,"successful":1,"failed":0},"aapc.log-test":{"total":10,"successful":5,"failed":0},"shipapidev-":{"total":10,"successful":5,"failed":0},".tasks":{"total":1,"successful ":1,"failed":0},"aapc.log-navigators":{"total":10,"successful":5,"failed":0},".security":{"total":1,"successful":1,"failed":0},"searchguard":{"total":1,"successful":1,"failed":0} ,".kibana":{"total":2,"successful":1,"failed":0},"aapc.log-stage":{"total":10,"successful":5,"failed":0},".kibana6_9":{"total":1,"successful":1,"failed":0},"aapc.log-local":{"tot al":10,"successful":5,"failed":0},".kibana6_8":{"total":1,"successful":1,"failed":0},".kibana6_7":{"total":1,"successful":1,"failed":0},"wwwdev-":{"total":10,"successful":5,"fail ed":0}}')*

If I read this right, the error is during the close section of action.yml. It never gets to the delete at all.

So this 409 error is indicating that you're trying to close aapc.log-prod, which could not perform a synced flush because of pending operations, which suggests that new data is still trying to be indexed/added.

Are you sure you want to close an index which is actively receiving data?

Probably not - but the operation wouldn't try to close it unless it was more than 30 days old. Correct? Is this typical? How can I push the pending operations through? or exclude aapc.log-prod somehow?

If there are pending operations and I don't close the indices before I run the delete-indices action, what happens?

There is no need to close an index before deleting it. You can delete without closing.

this took care of my issue. Thank You!