# Failed migration of system indices for the watcher and watcher not firing after update

**URL:** https://discuss.elastic.co/t/failed-migration-of-system-indices-for-the-watcher-and-watcher-not-firing-after-update/350704
**Category:** Elasticsearch
**Created:** [January 10, 2024, 7:49am UTC](https://discuss.elastic.co/t/failed-migration-of-system-indices-for-the-watcher-and-watcher-not-firing-after-update/350704 "2024-01-10T07:49:32Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![AEA27](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aea27/32/124389_2.png) [@AEA27](https://discuss.elastic.co/u/AEA27)
#### Post date: [January 10, 2024, 7:49am UTC](https://discuss.elastic.co/t/failed-migration-of-system-indices-for-the-watcher-and-watcher-not-firing-after-update/350704/1 "2024-01-10T07:49:32Z")

</div>

continuation of [Failed migration of system indices (.triggered\_watches) with Upgrade Assistant using v7.17.14](https://discuss.elastic.co/t/failed-migration-of-system-indices-triggered-watches-with-upgrade-assistant-using-v7-17-14/346966)

Summary, upgrade assistant was giving an error for the watcher when migrating system indices.

I tried deleting the index template but the error persists.

Even when I try getting the index template I get a 404.

When I send this request `GET /_index_template/.watches` I get

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "resource_not_found_exception",
        "reason" : "index template matching [.watches] not found"
      }
    ],
    "type" : "resource_not_found_exception",
    "reason" : "index template matching [.watches] not found"
  },
  "status" : 404
}

```

After that I decided to delete all the watchers since they were not getting triggered anymore anyway since I updated to version `7.17.14` from version `7.12.0`

```auto
[instance-0000000013] unable to create new index [.watches-reindexed-for-8] from feature [watcher] because it would match legacy templates [[.watches]]
[instance-0000000013] task [upgrade-system-indices] failed with an exception java.lang.IllegalStateException: unable to create new index [.watches-reindexed-for-8] because it would match legacy templates [[.watches]] at org.elasticsearch.upgrades.SystemIndexMigrator.migrateSingleIndex(SystemIndexMigrator.java:418) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.upgrades.SystemIndexMigrator.lambda$run$3(SystemIndexMigrator.java:213) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.upgrades.SystemIndexMigrator.lambda$updateTaskState$11(SystemIndexMigrator.java:359) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192) [x-pack-security-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$DelegatingActionListener.onResponse(ActionListener.java:186) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.persistent.PersistentTasksClusterService$4.clusterStateProcessed(PersistentTasksClusterService.java:267) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.clusterStateProcessed(MasterService.java:645) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$processedDifferentClusterState$1(MasterService.java:517) [elasticsearch-7.17.14.jar:7.17.14] at java.util.ArrayList.forEach(ArrayList.java:1596) [?:?] at org.elasticsearch.cluster.service.MasterService$TaskOutputs.processedDifferentClusterState(MasterService.java:517) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.onPublicationSuccess(MasterService.java:322) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:314) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:287) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService.access$100(MasterService.java:63) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:170) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262) [elasticsearch-7.17.14.jar:7.17.14] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225) [elasticsearch-7.17.14.jar:7.17.14] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] at java.lang.Thread.run(Thread.java:1583) [?:?]

```

Anything I can do now?

---

<div class="post-metadata">

### Author: ![William\_Brafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/william_brafford/32/51559_2.png) [@William\_Brafford](https://discuss.elastic.co/u/William_Brafford)
#### Post date: [January 16, 2024, 7:38pm UTC](https://discuss.elastic.co/t/failed-migration-of-system-indices-for-the-watcher-and-watcher-not-firing-after-update/350704/2 "2024-01-16T19:38:03Z")

</div>

I think I see the problem. The error message indicates that `.watches` is a deprecated _legacy_ template, so you need to get or delete the template using different endpoints. (Docs: [Get index template (legacy)](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/indices-get-template-v1.html), [Delete Index Template (legacy)](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/indices-delete-template-v1.html))

```auto
# view the template
GET /_template/.watches

# delete the template
DELETE /_template/.watches

```

I hope this helps you proceed with the upgrade.

---

<div class="post-metadata">

### Author: ![AEA27](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aea27/32/124389_2.png) [@AEA27](https://discuss.elastic.co/u/AEA27)
#### Post date: [January 17, 2024, 7:45am UTC](https://discuss.elastic.co/t/failed-migration-of-system-indices-for-the-watcher-and-watcher-not-firing-after-update/350704/3 "2024-01-17T07:45:14Z")

</div>

It worked! Thank you very much!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 14, 2024, 7:45am UTC](https://discuss.elastic.co/t/failed-migration-of-system-indices-for-the-watcher-and-watcher-not-firing-after-update/350704/4 "2024-02-14T07:45:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
