Can't delete ML jobs which was on hard_limit

Elastic Stack Elasticsearch
1

Hi, This is Jason.

I'm in trouble for deleting hard-limited ml-job.

image
image.png1678×75 7.81 KB

I'd deleted them yesterday, however It's still deleting :frowning:
I used dev-tool to delete them again today, but failed like below

image
image.png730×586 27.8 KB

How can I delete these jobs?

Thanks & regards
Jason

2

I suspect that because these jobs were "too big" for your setup (hence giving you a hard_limit) then the deleting of these jobs (which also has to remove docs from some of the .ml-* indices) is also very taxing on your system.

First, some obvious questions:

  1. What version are you on?
  2. How many jobs have you configured?
  3. How many are actively running?
  4. If you were to invoke the following API call in DevTools console several times:

GET _cat/indices/.ml-*?v

and look at the docs.count and docs.deleted columns, do you notice if those numbers are changing significantly? In other words, is there an active background task that is still trying to delete stuff?

1 Like
3
  1. What version are you on?
    - v6.8.0
  2. How many jobs have you configured?
    - 27 jobs total, 26 running jobs and 25 datafeeds
  3. How many are actively running?
    - 26 jobs running
  4. If you were to invoke the following API call in DevTools console several times:
    - yes about 3-5 times

I saw this morning the deprecated jobs have been deleted completly. It took 4 days for deleteing, wow.
I think they were too big.

Thanks fot the tip

GET _cat/indices/.ml-*?v

and look at the docs.count and docs.deleted columns, do you notice if those >numbers are changing significantly? In other words, is there an active background task >that is still trying to delete stuff?

1 Like
4

Since your are on v6.8.0 you might also want to consider the following:

DELETE _ml/_delete_expired_data

This request normally occurs every night with our nightly maintenance, but there is a bug in our nightly maintenance scheduler. The bug is fixed in 6.8.4 and 7.4.1. But, since this bug exists in your version, your jobs will continually accumulate results and state without bound.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.