Curator Reindex

Elastic Stack Elasticsearch
1

Hi Everyone,

Today i wanted to manage my reindex from daily indices to monthly with Curator.

The thing is i encoutered a error and i can't debug it, waste a bit time on google to find some help but nothing.

Here Action.yml

---
#################################
#
# Curator - Reindex Daily To Monthly
#
#################################

# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.

actions:
  1:
    action: reindex
    description: Curator - Reindex Daily To Monthly
    options:
      max_wait: -1
      wait_interval: 10
      wait_for_completion: True
      ignore_empty_list: False
      continue_if_exception: False
      disable_action: False
      request_body:
        source:
          index: REINDEX_SELECTION
        dest:
          index: 'toto-reindex'
    filters:
    - filtertype: pattern
      kind: prefix
      value: toto-2018.03.
# End of Curator - Reindex Daily To Monthly

And the return when executing :

2018-03-16 17:12:17,487 ERROR curator.actions.reindex _post_run_quick_check:1266 The index described as "toto-reindex" was not found after the reindex operation. Check Elasticsearch logs for more information.
2018-03-16 17:12:17,487 ERROR curator.cli run:184 Failed to complete action: reindex. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: Reindex failed. The index or alias identified by "toto-reindex" was not found.

2

I am not really sure if Curator can do such operation. Maybe @theuntergeek could clarify here.

But another way to do what you want is to call the Reindex API directly with a proper script. The following following can be used as a base script to work on https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#_reindex_daily_indices

3

Is dynamic index creation disabled in your cluster? This would definitely be the result if that were the case. Elasticsearch would also have logs explaining what happened with your reindex.

The error you're getting from Curator is simply telling you that Curator attempted to perform the action, and something happened. Curator does a sanity check after the reindex to make sure the index is at least present. It isn't, so Curator is raising the red flag here. I would like to see the Curator log messages preceding the error lines you shared. There may be more information to be obtained within those lines.

4

Yes, it can, and a whole lot more. See the request_body documentation.

1 Like
5

Hi dudes, thanks for replying. @theuntergeek & @thiago

Here lines before those i already put in the post : (Not the same date)

moving from list.
2018-03-19 08:22:37,617 DEBUG              curator.utils        iterate_filters:1087 Post-instance: ['toto-2018.03.19', 'toto-2018.03.20']
2018-03-19 08:22:37,617 DEBUG     curator.actions.reindex               __init__:1055 REQUEST_BODY = {'source': {'index': 'REINDEX_SELECTION'}, 'dest': {'index': 'toto-reindex'}}
2018-03-19 08:22:37,617 DEBUG     curator.actions.reindex               __init__:1228 Reindexing indices: ['toto-2018.03.19', 'toto-2018.03.20']
2018-03-19 08:22:37,617 DEBUG                curator.cli         process_action:99   Doing the action here.
2018-03-19 08:22:37,617 INFO      curator.actions.reindex              do_action:1335 Commencing reindex operation
2018-03-19 08:22:37,617 DEBUG     curator.actions.reindex              do_action:1337 REINDEX: request body: {'source': {'index': ['toto-2018.03.19', 'toto-2018.03.20']}, 'dest': {'index': 'toto-reindex'}} with arguments: refresh=True requests_per_second=-1 slices=1 timeout=60s wait_for_active_shards=1 wait_for_completion=True
2018-03-19 08:22:37,622 DEBUG     curator.actions.reindex              do_action:1341 TASK ID = COAvg19PRzmJS-VVcpZTag:25857014
2018-03-19 08:22:37,623 DEBUG              curator.utils            wait_for_it:1725 Elapsed time: 0 seconds
2018-03-19 08:22:37,625 DEBUG              curator.utils             task_check:1623 running_time_in_nanos = 0.0038247050000000003
2018-03-19 08:22:37,625 DEBUG              curator.utils             task_check:1635 Full Task Data: {'completed': False, 'task': {'node': 'COAvg19PRzmJS-VVcpZTag', 'id': 25857014, 'type': 'transport', 'action': 'indices:data/write/reindex', 'status': {'total': 0, 'updated': 0, 'created': 0, 'deleted': 0, 'batches': 0, 'version_conflicts': 0, 'noops': 0, 'retries': {'bulk': 0, 'search': 0}, 'throttled_millis': 0, 'requests_per_second': -1.0, 'throttled_until_millis': 0}, 'description': 'reindex from [toto-2018.03.19, toto-2018.03.20] to [toto-reindex]', 'start_time_in_millis': 1521444157620, 'running_time_in_nanos': 3824705, 'cancellable': True}}
2018-03-19 08:22:37,625 INFO               curator.utils             task_check:1638 Task "reindex from [toto-2018.03.19, toto-2018.03.20] to [toto-reindex]" with task_id "COAvg19PRzmJS-VVcpZTag:25857014" has been running for 0.0038247050000000003 seconds
2018-03-19 08:22:37,625 DEBUG              curator.utils            wait_for_it:1728 Response: False
2018-03-19 08:22:37,625 DEBUG              curator.utils            wait_for_it:1748 Action "reindex" not yet complete, 0 total seconds elapsed. Waiting 10 seconds before checking again.
2018-03-19 08:22:47,633 DEBUG              curator.utils            wait_for_it:1725 Elapsed time: 10 seconds
2018-03-19 08:22:47,637 DEBUG              curator.utils             task_check:1623 running_time_in_nanos = 0.0050319530000000005
2018-03-19 08:22:47,637 INFO               curator.utils             task_check:1631 Task "reindex from [toto-2018.03.19, toto-2018.03.20] to [toto-reindex]" completed at 2018-03-19T08:22:37Z.
2018-03-19 08:22:47,637 DEBUG              curator.utils            wait_for_it:1728 Response: True
2018-03-19 08:22:47,637 DEBUG              curator.utils            wait_for_it:1733 Action "reindex" finished executing (may or may not have been successful)
2018-03-19 08:22:47,637 DEBUG              curator.utils            wait_for_it:1751 Result: True

Logs after filter part (filters are too long for the topic and useless i think)

So i hope this can help you to resolve my issue with me.

Edit: When adding action create index it seems work fine, so it could solve my issue.

Edit 2 : Does Curator Reindex action support Date Math in Dest/Source ?
-> Creating index with dath math and reindex with same date math for example

Edit 3 : All done

6

You marked it "solved." Is it solved?

I can tell you that the task says it completed pretty much instantaneously:

2018-03-19 08:22:37,617 INFO      curator.actions.reindex              do_action:1335 Commencing reindex operation
...
2018-03-19 08:22:47,637 INFO               curator.utils             task_check:1631 Task "reindex from [toto-2018.03.19, toto-2018.03.20] to [toto-reindex]" completed at 2018-03-19T08:22:37Z.

It made you wait 10 seconds, just to be sure (mostly so the Tasks API has a chance to spool up and actually register the Reindex "task"), but it says it completed at exactly the moment it started. The only way that happens is if the indices basically had one or two docs is all, or there's something else wrong on the Elasticsearch side. Let me run through the program flow a bit.

After selecting the indices you want reindexed, all Curator does is initiate the Reindex API call. This creates a "task" which is viewable in the Tasks API. Curator monitors the Tasks API for completion of the Reindex task it initiated. When it gets the "completed" response back from the Tasks API, it checks to see if the index you set for dest is, in fact, created (Curator does no size or doc count checking). But that's what Curator does. The problem for you seems to be on the Elasticsearch side. Why do I suggest this? Because you'd have a similar problem if you made the exact same API calls yourself, manually.

Your next step is to look through the Elasticsearch logs (particularly the master nodes, and/or the node that Curator is connecting to) at around 2018-03-19 08:22:37,617 to see what is going on in Elasticsearch, since Curator is doing what it is supposed to do.

You may also be able to dig up some information on task COAvg19PRzmJS-VVcpZTag:25857014 (which is the node the task ran on, a colon, and the id number assigned for the task) to see what the result was. Curator only checked to see if the task completed.

1 Like
7

Yeah i marked it solved, because i found a alternativ to my issue (Creating the reindex index in my action file)

Yes indeed, when i ran reindex manually got the same error

Maybe it could be those empty index that i created to test curator reindex ?

I will check out logs to see if somethings wrong

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.