# How to solve hard and soft limit machine learning jobs

**URL:** <https://discuss.elastic.co/t/how-to-solve-hard-and-soft-limit-machine-learning-jobs/268933>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [March 31, 2021, 3:50pm UTC](https://discuss.elastic.co/t/how-to-solve-hard-and-soft-limit-machine-learning-jobs/268933 "2021-03-31T15:50:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abdelhalim](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@Abdelhalim](https://discuss.elastic.co/u/Abdelhalim)\
**Post date:** [March 31, 2021, 3:50pm UTC](https://discuss.elastic.co/t/how-to-solve-hard-and-soft-limit-machine-learning-jobs/268933/1 "2021-03-31T15:50:58Z")

</div>

Hello;

I have created a machine learning job to detect port scanner using the `packetbeat-*` index.

**type of job:** Population  
**Population field:** destination.ip  
**metric:** distinct count(destination port)  
**influencers:** destination.ip and source.ip  
**bukcet span:** 15min

it's working perfectly, but I am getting warning of `hard limits` AND `soft_limit`,

```auto
Job memory status changed to soft_limit; memory pruning will now be more aggressive
Job memory status changed to hard_limit; job exceeded model memory limit 23mb by 1.7mb. Adjust the analysis_limits.model_memory_limit setting to ensure all data is analyzed

```

I have a dedicated machine learninig node: **6 CPU** and **8Go RAM**

Could you please tell me how can I solve this warninigs !

Thanks for your help !

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [March 31, 2021, 4:27pm UTC](https://discuss.elastic.co/t/how-to-solve-hard-and-soft-limit-machine-learning-jobs/268933/2 "2021-03-31T16:27:01Z")

</div>

Except that it is not working perfectly - the job is "throwing away" data in order to keep the job from using too much memory.

By default, only 30% of your 8GB node (i.e. 2.4GB) is given for ML to use (see `xpack.ml.max_machine_memory_percent` at [Machine learning settings in Elasticsearch | Elasticsearch Reference [7.12] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-settings.html))

You should look to see how much memory is being used by other jobs (summing up the `model_bytes` for all running/open jobs) using

`GET _ml/anomaly_detectors/_stats`

If there's enough room, you can increase the memory limit on this job. See `model_memory_limit` at [https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-update-job.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-update-job.html)

If not, you need a bigger ML node.

---

<div class="post-metadata">

**Author:** ![Abdelhalim](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@Abdelhalim](https://discuss.elastic.co/u/Abdelhalim)\
**Post date:** [April 1, 2021, 9:28am UTC](https://discuss.elastic.co/t/how-to-solve-hard-and-soft-limit-machine-learning-jobs/268933/3 "2021-04-01T09:28:18Z")

</div>

Thanks for your explanation @richcollier  
Now I understand better how it works

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 29, 2021, 9:28am UTC](https://discuss.elastic.co/t/how-to-solve-hard-and-soft-limit-machine-learning-jobs/268933/4 "2021-04-29T09:28:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.