continuation of Failed migration of system indices (.triggered_watches) with Upgrade Assistant using v7.17.14
Summary, upgrade assistant was giving an error for the watcher when migrating system indices.
I tried deleting the index template but the error persists.
Even when I try getting the index template I get a 404.
When I send this request GET /_index_template/.watches I get
{
"error" : {
"root_cause" : [
{
"type" : "resource_not_found_exception",
"reason" : "index template matching [.watches] not found"
}
],
"type" : "resource_not_found_exception",
"reason" : "index template matching [.watches] not found"
},
"status" : 404
}
After that I decided to delete all the watchers since they were not getting triggered anymore anyway since I updated to version 7.17.14 from version 7.12.0
[instance-0000000013] unable to create new index [.watches-reindexed-for-8] from feature [watcher] because it would match legacy templates [[.watches]]
[instance-0000000013] task [upgrade-system-indices] failed with an exception java.lang.IllegalStateException: unable to create new index [.watches-reindexed-for-8] because it would match legacy templates [[.watches]] at org.elasticsearch.upgrades.SystemIndexMigrator.migrateSingleIndex(SystemIndexMigrator.java:418) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.upgrades.SystemIndexMigrator.lambda$run$3(SystemIndexMigrator.java:213) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.upgrades.SystemIndexMigrator.lambda$updateTaskState$11(SystemIndexMigrator.java:359) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192) [x-pack-security-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$DelegatingActionListener.onResponse(ActionListener.java:186) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.persistent.PersistentTasksClusterService$4.clusterStateProcessed(PersistentTasksClusterService.java:267) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.clusterStateProcessed(MasterService.java:645) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$processedDifferentClusterState$1(MasterService.java:517) [elasticsearch-7.17.14.jar:7.17.14] at java.util.ArrayList.forEach(ArrayList.java:1596) [?:?] at org.elasticsearch.cluster.service.MasterService$TaskOutputs.processedDifferentClusterState(MasterService.java:517) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.onPublicationSuccess(MasterService.java:322) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:314) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:287) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225) [elasticsearch-7.17.14.jar:7.17.14] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] at java.lang.Thread.run(Thread.java:1583) [?:?]
Anything I can do now?