# Machine learning error: persistent task is awating node assignment **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [November 24, 2021, 10:57am UTC](https://discuss.elastic.co/t/machine-learning-error-persistent-task-is-awating-node-assignment/290043 "2021-11-24T10:57:03Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Post date:** [November 24, 2021, 10:57am UTC](https://discuss.elastic.co/t/machine-learning-error-persistent-task-is-awating-node-assignment/290043/1 "2021-11-24T10:57:03Z")
Hello, I hope this message finds Elastic community and their loved ones safe and healthy. I have a three node cluster: 1. Node-1 and Node-2 hold data & carry out computation: - both have has 24 GB of RAM with jvm set to 12 GB - 3 vCPUs @ 3.4 GHz per vCPU respectively. 1. All of the node's are running on **separate SSDs** and IOPS are under 50% even at peak load. 2. Average CPU usage for node-1 and node-2 is ~50% for the last 7 days. 3. Average Memory utilisation is ~30% for the last 7 days. (JVM heap is around ~55%) 4. Node-3 is a voting only node. For all of my ML jobs I get `persistent task is awaiting node assignment`. Do I need to spin up a dedicated ML node? or can I assign to the current nodes? ---
**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\ **Post date:** [November 24, 2021, 11:33am UTC](https://discuss.elastic.co/t/machine-learning-error-persistent-task-is-awating-node-assignment/290043/2 "2021-11-24T11:33:09Z")
Which version are you running? Are node-1 and node-2 general purpose nodes with all roles? In other words, you haven't put `node.ml: false` or explicitly specified `node.roles` in either of their `elasticsearch.yml` files? If those nodes have all the roles, including the ML role then you don't necessarily have to spin up a dedicated node. I am wondering if you are suffering from the poor communication outlined in [[ML] Make the job notifications related to node assignment more user friendly · Issue #79270 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/79270). Often when ML jobs cannot be assigned the reason is transitory and they _do_ get assigned quickly afterwards. But we show warnings for 24 hours which make people think there's a permanent problem. Where exactly are you seeing `persistent task is awaiting node assignment`? And are _any_ of your jobs running? If some jobs are running but not others after you restarted your cluster then it could be that the cluster was big enough to run all the jobs originally but isn't now after the jobs saw a lot of data to model and increased in size. If that's what's happened it will be possible to tell from the output of the [Get anomaly detection job statistics API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-job-stats.html). ---
**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Post date:** [November 25, 2021, 4:33am UTC](https://discuss.elastic.co/t/machine-learning-error-persistent-task-is-awating-node-assignment/290043/3 "2021-11-25T04:33:36Z")
Hello @droberts195, thank you very much for your reply. ```auto Edit: 1. I am running Elastic 7.15.2 with platinum trial license ``` 1. For **nodes 1 & 2** following options are set: ```auto node.master: true node.voting_only: false ``` 1. When I create custom ML jobs they do run and finish, but I can't seem to keep them always "on". Here is snapshot of ML page where I see a different error too & ML has 2 nodes as per the status page. ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/e/ee6cf9654376ce34f1c1258c8c5261ea489233c3.png) 1. When I start prebuilt jobs such as those part of Elastic Security, they keep shutting down abruptly. I've started jobs again and I will check how they continue: ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/4/446ea9e57a122932eda9fb8bc8688a20d9cd131e.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/1/c1bcd63105783babe384795072d9c9645a6485e6.png) 1. Some custom jobs end with "hard limit" in the memory, here is the output of their status. What do I need to change here? ```auto { "count" : 1, "jobs" : [ { "job_id" : "dns_rare_oxford", "data_counts" : { "job_id" : "dns_rare_oxford", "processed_record_count" : 486569375, "processed_field_count" : 486569375, "input_bytes" : 41188106693, "input_field_count" : 486569375, "invalid_date_count" : 0, "missing_field_count" : 0, "out_of_order_timestamp_count" : 0, "empty_bucket_count" : 0, "sparse_bucket_count" : 8, "bucket_count" : 18, "earliest_record_timestamp" : 1632442643000, "latest_record_timestamp" : 1632508200000, "last_data_time" : 1635241410029, "latest_sparse_bucket_timestamp" : 1632484800000, "input_record_count" : 486569375, "log_time" : 1635241410029 }, "model_size_stats" : { "job_id" : "dns_rare_oxford", "result_type" : "model_size_stats", "model_bytes" : 44051824, "peak_model_bytes" : 44051824, "model_bytes_exceeded" : 20890, "model_bytes_memory_limit" : 44040192, "total_by_field_count" : 11732, "total_over_field_count" : 0, "total_partition_field_count" : 2, "bucket_allocation_failures_count" : 18, "memory_status" : "hard_limit", "assignment_memory_basis" : "model_memory_limit", "categorized_doc_count" : 0, "total_category_count" : 0, "frequent_category_count" : 0, "rare_category_count" : 0, "dead_category_count" : 0, "failed_category_count" : 0, "categorization_status" : "ok", "log_time" : 1635241417030, "timestamp" : 1632502800000 }, "forecasts_stats" : { "total" : 0, "forecasted_jobs" : 0 }, "state" : "closed", "timing_stats" : { "job_id" : "dns_rare_oxford", "bucket_count" : 18, "total_bucket_processing_time_ms" : 316.0, "minimum_bucket_processing_time_ms" : 4.0, "maximum_bucket_processing_time_ms" : 104.0, "average_bucket_processing_time_ms" : 17.555555555555557, "exponential_average_bucket_processing_time_ms" : 6.193679389034066, "exponential_average_bucket_processing_time_per_hour_ms" : 28.0 } } ] } ``` Please do let me know if more information is required. ---
**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\ **Post date:** [November 25, 2021, 9:54am UTC](https://discuss.elastic.co/t/machine-learning-error-persistent-task-is-awating-node-assignment/290043/4 "2021-11-25T09:54:43Z")
1. The default for `node.ml` is `true`, so nodes 1 and 2 are ML nodes as well as master nodes. So there's nothing in the `elasticsearch.yml` settings that would cause a problem for what you are doing. 2. The jobs in the screenshot are in the `failed` state. Something is going wrong with them. The clue will be in the log files of the node they were running on at the time when they failed. If you search for the word `failed` in your logs then the other messages around that area will probably indicate why one of the jobs failed. If it's too low level to understand and doesn't contain any confidential information you could paste an extract of the log around the time of an ML job failure in this thread and I will have a look. 3. The "5 jobs unavailable" in the security app screenshot corresponds to the 5 failed jobs in the ML app screenshot. It's the same problem. We need to work out what's making them fail. 4. For the jobs that went to `hard_limit`, you would need to increase `{ "analysis_limits" : { "model_memory_limit" }}` - search for `model_memory_limit` in [Update anomaly detection jobs API | Elasticsearch Guide [7.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-update-job.html) for more information. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 23, 2021, 9:54am UTC](https://discuss.elastic.co/t/machine-learning-error-persistent-task-is-awating-node-assignment/290043/5 "2021-12-23T09:54:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.