# Error while delete ml anomaly job

**URL:** <https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [August 5, 2020, 3:02pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888 "2020-08-05T15:02:56Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [August 5, 2020, 3:02pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/1 "2020-08-05T15:02:56Z")

</div>

I'm trying to delete an ML job.

I used Kibana to do so, and Kibana shows "deleting" status for this job.

`DELETE _ml/anomaly_detectors/jobid?force=true`

gives:

> "reason" : {  
> "type" : "node\_disconnected\_exception",  
> "reason" : "[instance-0000000057][10.x.x.x:19998][indices:data/read/search[phase/query]] disconnected"  
> }

(node is up and running)

`POST _ml/anomaly_detectors/jobid/_close?force=true`

gives:

> No known job with id

Any assistance will be appreciated

---

<div class="post-metadata">

**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\
**Post date:** [August 5, 2020, 3:41pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/2 "2020-08-05T15:41:18Z")

</div>

What does delete without force return?

```auto
DELETE _ml/anomaly_detectors/jobid

```

How about just getting the job?

```auto
GET _ml/anomaly_detectors/jobid

```

Which version of Elasticsearch are you running?

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [August 5, 2020, 3:57pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/3 "2020-08-05T15:57:04Z")

</div>

Hi

Delete without force returns the same

GET...  
returns

```
{
  "error" : {
    "root_cause" : [],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : []
  },
  "status" : 503
}
```

---

<div class="post-metadata">

**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\
**Post date:** [August 5, 2020, 4:15pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/4 "2020-08-05T16:15:46Z")

</div>

It sounds like you might have lost your `.ml-config` index, or the node that index is on is inaccessible.

Which version of Elasticsearch are you using?

Does this show anything wrong?

```auto
GET _cat/indices/.ml*?v

```

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [August 5, 2020, 4:35pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/5 "2020-08-05T16:35:43Z")

</div>

> [@droberts195](#):
>
> `GET _cat/indices/.ml*?v`

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/f/efe4ba8bb814369d8ae5517b8288d48159d1d9d6.png)

version 7.8.1

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 5, 2020, 4:42pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/6 "2020-08-05T16:42:00Z")

</div>

@liorg2 - Do you have an active support contract with us? I'm thinking it would be more helpful if you could send in a support diagnostic bundle instead of this back-and-forth

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [August 5, 2020, 7:19pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/7 "2020-08-05T19:19:40Z")

</div>

unfortunately not...  
tried a restart and still the same

"type" : "search\_phase\_execution\_exception",  
"reason" : "all shards failed",  
"phase" : "query",  
"grouped" : true,  
"failed\_shards" : [  
{  
"shard" : 0,  
"index" : ".ml-anomalies-shared",  
"node" : "x4BPgr-OTuSr6CycZY2SuQ",  
"reason" : {  
"type" : "node\_disconnected\_exception",  
"reason" : "[instance-..........][10........:19662][indices:data/read/search[phase/query]] disconnected"  
}  
}  
],

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [August 5, 2020, 7:22pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/8 "2020-08-05T19:22:15Z")

</div>

now new error

"reason" : "No shard available for [get [.ml-config][\_doc][anomaly\_detector-..........-job]: routing [null]]"

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 5, 2020, 8:17pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/9 "2020-08-05T20:17:31Z")

</div>

> [@liorg2](#):
>
> unfortunately not...

I assume this is your answer to my question about having a support contract with us?

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [August 5, 2020, 8:18pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/10 "2020-08-05T20:18:43Z")

</div>

Yes.  
Btw what i did today is changing the ml node from 1 az into 2.

---

<div class="post-metadata">

**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\
**Post date:** [August 6, 2020, 9:25am UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/11 "2020-08-06T09:25:24Z")

</div>

I wonder if there is a problem with communications between nodes where the master node can communicate with all the other nodes in the cluster (hence `_cat/indices` shows no problems) but some of the other nodes cannot communicate with each other.

You talk about availability zones. Are you running in Elastic Cloud? Or have you set up your own cluster in the cloud?

One more thing you could try is to get the allocation explanations for the problematic indices - maybe that will show up something interesting:

```auto
GET /_cluster/allocation/explain?include_yes_decisions=true
{
  "index": ".ml-config",
  "shard": 0,
  "primary": true
}

GET /_cluster/allocation/explain?include_yes_decisions=true
{
  "index": ".ml-config",
  "shard": 0,
  "primary": false
}

GET /_cluster/allocation/explain?include_yes_decisions=true
{
  "index": ".ml-anomalies-shared",
  "shard": 0,
  "primary": true
}

GET /_cluster/allocation/explain?include_yes_decisions=true
{
  "index": ".ml-anomalies-shared",
  "shard": 0,
  "primary": false
}

```

One more problem I can see from your `_cat/indices` output is that you've ended up with a concrete index that should have been an alias. `.ml-anomalies-.write-test_93` should have been an alias. We have recently added [protection to stop this problem occurring](https://github.com/elastic/elasticsearch/pull/60315) but since you only have one document in that index it would be interesting to know what it is. Please could you do a search of `.ml-anomalies-.write-test_93` and let us know what document is in it? You can redact anything confidential - just seeing the field names of the document that's in there will help us understand why it got created. Then you can delete that index and, if the `test_93` job still exists, create `.ml-anomalies-.write-test_93` as an alias of `.ml-anomalies-shared`. (If the `test_93` job no longer exists then just delete that index.)

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 6, 2020, 2:32pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/12 "2020-08-06T14:32:11Z")

</div>

> [@liorg2](#):
>
> Yes

It looks like you're a [customer of Elastic Cloud](https://discuss.elastic.co/t/elastic-cloud-availability-zones/243859) and with that comes support. And, if you're experiencing inter-node communications problems, this is exactly the kind of thing support should help you with.

If you don't know how to access support, DM me the email address you use for your cloud account and we'll get someone to help you.

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [August 6, 2020, 4:27pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/13 "2020-08-06T16:27:11Z")

</div>

Hey, thanks a lot for the help.

I tried to delete the jobs right now sing the api and it worked well 🙂

---

<div class="post-metadata">

**Author:** ![liorg2](https://avatars.discourse-cdn.com/v4/letter/l/ed8c4c/32.png) [@liorg2](https://discuss.elastic.co/u/liorg2)\
**Post date:** [August 6, 2020, 4:29pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/14 "2020-08-06T16:29:52Z")

</div>

Thanks a lot @droberts195  
Yes it's an old job. I'll delete this index

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 3, 2020, 4:30pm UTC](https://discuss.elastic.co/t/error-while-delete-ml-anomaly-job/243888/15 "2020-09-03T16:30:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.