Failed migration of system indices (.triggered_watches) with Upgrade Assistant using v7.17.14

Elastic Stack Elasticsearch
1

Using the upgrade assistant the migration of system indices fails for watcher which we're using. We're using elastic cloud and trying to migrate to Elasticsearch 8. Another fun fact is that the index that is failing to reindex is empty.

GET .triggered_watches/_count

returns

#! this request accesses system indices: [.triggered_watches], but in a future major version, direct access to system indices will be prevented by default
{
  "count" : 0,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

The error message is the following

[instance-0000000013] unable to create new index [.triggered_watches-reindexed-for-8] from feature [watcher] because it would match composable template [.triggered_watches]
[instance-0000000013] task [upgrade-system-indices] failed with an exception java.lang.IllegalStateException: unable to create new index [.triggered_watches-reindexed-for-8] because it would match composable template [.triggered_watches] at org.elasticsearch.upgrades.SystemIndexMigrator.migrateSingleIndex(SystemIndexMigrator.java:399) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.upgrades.SystemIndexMigrator.lambda$run$3(SystemIndexMigrator.java:213) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.upgrades.SystemIndexMigrator.lambda$updateTaskState$11(SystemIndexMigrator.java:359) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192) [x-pack-security-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$DelegatingActionListener.onResponse(ActionListener.java:186) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.persistent.PersistentTasksClusterService$4.clusterStateProcessed(PersistentTasksClusterService.java:267) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.clusterStateProcessed(MasterService.java:645) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$processedDifferentClusterState$1(MasterService.java:517) [elasticsearch-7.17.14.jar:7.17.14] at java.util.ArrayList.forEach(ArrayList.java:1596) [?:?] at org.elasticsearch.cluster.service.MasterService$TaskOutputs.processedDifferentClusterState(MasterService.java:517) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.onPublicationSuccess(MasterService.java:322) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:314) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:287) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225) [elasticsearch-7.17.14.jar:7.17.14] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] at java.lang.Thread.run(Thread.java:1583) [?:?]
2

I'm sorry you're hitting this problem. I suggest deleting the composable template (system indices no longer use templates, but such templates may be left over from previous versions), and you should be able to proceed with the upgrade.

It should also be fine to delete an empty .triggered_watches index.

3

Thanks for the quick response.

It solved the issue for .triggered)watches but now we're facing the same issue with .watches, unfortunately we have a few "watches" so we can't just delete the index, what would you recommend to do in this kind of situations?

Thanks

4

Sorry for taking a long time to respond. I was on vacation.

If you are still getting a message like this one, but with .watches instead of .triggered_watches, you can delete the

You can delete the index template without affecting the index at all. In this case, you'd have something like

DELETE /_index_template/.watches,.triggered_watches

These templates were created before Elasticsearch had the "system index" feature, and Elasticsearch now uses a different internal mechanism to create settings and mappings for .watches and .triggered_watches.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.