# ML Job hard limit error on Linux

**URL:** <https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [October 3, 2018, 11:00am UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865 "2018-10-03T11:00:08Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![TamasK](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@TamasK](https://discuss.elastic.co/u/TamasK)\
**Post date:** [October 3, 2018, 11:00am UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/1 "2018-10-03T11:00:08Z")

</div>

Hi,

Maybe somebody can help me to solve the following problem

Problem description: If I create and start new multimetric job, then memory status always changes to **hard limit**. (This job worked on Windows without any problem)

OS: Ubuntu 16.04.5 LTS (GNU/Linux 4.15.18-1-pve x86\_64)  
Memory: 12GB  
Elastic: 6.4.1  
Java: Oracle 1.8.0\_181-b13 (x64)  
Heap size: 4GB  
vm.max\_map\_count=262144

Index:  
Docs Count: 8431  
Storage Size: 2.2mb

Job:  
established\_model\_memory:143.2 KB  
model\_memory\_limitI I tried different values from 12MB to 1200MB

**job error message:**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/7/27b8832b9c7d30d3b7a77888b4f0e6e29af12fd2.png)

> Job memory status changed to hard\_limit at 83.7kb; adjust the analysis\_limits.model\_memory\_limit setting to ensure all data is analyzed.

If I create another job on a same but bigger index (with 3 millon document) then the limit value is 69mb in the error message.

I didn't see any error message in the elastic log.

**Error was replicated on another linux machine.**

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [October 3, 2018, 2:46pm UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/2 "2018-10-03T14:46:51Z")

</div>

Can you provide the following info on your ML Job?

```auto
GET _xpack/ml/anomaly_detectors/yourjobnamehere/_stats?pretty

```

---

<div class="post-metadata">

**Author:** ![TamasK](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@TamasK](https://discuss.elastic.co/u/TamasK)\
**Post date:** [October 3, 2018, 2:55pm UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/3 "2018-10-03T14:55:06Z")

</div>

> {  
> "count": 1,  
> "jobs": [  
> {  
> "job\_id": "dev2",  
> "data\_counts": {  
> "job\_id": "dev2",  
> "processed\_record\_count": 8431,  
> "processed\_field\_count": 16862,  
> "input\_bytes": 741326,  
> "input\_field\_count": 16862,  
> "invalid\_date\_count": 0,  
> "missing\_field\_count": 0,  
> "out\_of\_order\_timestamp\_count": 0,  
> "empty\_bucket\_count": 8,  
> "sparse\_bucket\_count": 1,  
> "bucket\_count": 431,  
> "earliest\_record\_timestamp": 1199059200000,  
> "latest\_record\_timestamp": 1459900800000,  
> "last\_data\_time": 1538492502740,  
> "latest\_empty\_bucket\_timestamp": 1451520000000,  
> "latest\_sparse\_bucket\_timestamp": 1388016000000,  
> "input\_record\_count": 8431  
> },  
> "model\_size\_stats": {  
> "job\_id": "dev2",  
> "result\_type": "model\_size\_stats",  
> "model\_bytes": 144616,  
> "total\_by\_field\_count": 5,  
> "total\_over\_field\_count": 0,  
> "total\_partition\_field\_count": 6,  
> "bucket\_allocation\_failures\_count": 398,  
> "memory\_status": "hard\_limit",  
> "log\_time": 1538492507000,  
> "timestamp": 1458777600000  
> },  
> "forecasts\_stats": {  
> "total": 0,  
> "forecasted\_jobs": 0  
> },  
> "state": "closed"  
> }  
> ]  
> }

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [October 3, 2018, 2:58pm UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/4 "2018-10-03T14:58:48Z")

</div>

Thanks - sorry to ask again, but actually I really wanted to get the full job details, so can you please rerun

```auto
GET _xpack/ml/anomaly_detectors/dev2/?pretty

```

(this is, without the `_stats` part)

---

<div class="post-metadata">

**Author:** ![TamasK](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@TamasK](https://discuss.elastic.co/u/TamasK)\
**Post date:** [October 3, 2018, 3:01pm UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/5 "2018-10-03T15:01:20Z")

</div>

> {  
> "count": 1,  
> "jobs": [  
> {  
> "job\_id": "dev2",  
> "job\_type": "anomaly\_detector",  
> "job\_version": "6.4.1",  
> "description": "",  
> "create\_time": 1538492492071,  
> "finished\_time": 1538492509358,  
> "established\_model\_memory": 144616,  
> "analysis\_config": {  
> "bucket\_span": "7d",  
> "detectors": [  
> {  
> "detector\_description": "non\_null\_sum(ve\_costamountactual)",  
> "function": "non\_null\_sum",  
> "field\_name": "ve\_costamountactual",  
> "partition\_field\_name": "ve\_itemno.keyword",  
> "detector\_index": 0  
> }  
> ],  
> "influencers": [  
> "ve\_itemno.keyword"  
> ]  
> },  
> "analysis\_limits": {  
> "model\_memory\_limit": "12mb",  
> "categorization\_examples\_limit": 4  
> },  
> "data\_description": {  
> "time\_field": "ve\_postingdate",  
> "time\_format": "epoch\_ms"  
> },  
> "model\_snapshot\_retention\_days": 1,  
> "custom\_settings": {  
> "created\_by": "multi-metric-wizard"  
> },  
> "model\_snapshot\_id": "1538492507",  
> "results\_index\_name": "shared"  
> }  
> ]  
> }

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [October 3, 2018, 3:04pm UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/6 "2018-10-03T15:04:50Z")

</div>

Thank you - can you also tell me the approximate cardinality of the field `ve_itemno.keyword`?

```auto
GET yourindexname/_search
{
  "size": 0,
  "aggs": {
    "cardinality": {
      "cardinality": {
        "field": "ve_itemno.keyword"
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![TamasK](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@TamasK](https://discuss.elastic.co/u/TamasK)\
**Post date:** [October 3, 2018, 3:06pm UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/7 "2018-10-03T15:06:31Z")

</div>

```
{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 8431,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cardinality": {
      "value": 5
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [October 3, 2018, 3:29pm UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/8 "2018-10-03T15:29:53Z")

</div>

Thanks for supplying the info - I'll discuss this with others internally and hopefully get back to you soon.

---

<div class="post-metadata">

**Author:** ![TamasK](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@TamasK](https://discuss.elastic.co/u/TamasK)\
**Post date:** [October 3, 2018, 4:48pm UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/9 "2018-10-03T16:48:00Z")

</div>

Some information, that can help your investigation. My original index has 3 million documents where ve\_itemno cardinality is 16141.

I wanted to create a job, that proccess only the small part of documents.

**1. solution attempt**

I created an advanced job and selected documents with a query

**2. solution attempt**

I created a new smaller index form the original with the reindex command and then I used a multimetric job.

Both solution worked on Windows, but failed on Linux

---

<div class="post-metadata">

**Author:** ![TamasK](https://avatars.discourse-cdn.com/v4/letter/t/65b543/32.png) [@TamasK](https://discuss.elastic.co/u/TamasK)\
**Post date:** [October 4, 2018, 7:39am UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/10 "2018-10-04T07:39:54Z")

</div>

A new small index was created normal way (PUT and logstash), but I received the same hard limit error message.

---

<div class="post-metadata">

**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\
**Post date:** [October 17, 2018, 10:42am UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/11 "2018-10-17T10:42:43Z")

</div>

We believe the bug here is that hard limit can be incorrectly triggered too soon when the bucket span is 1 day or longer. We have made [a change](https://github.com/elastic/ml-cpp/pull/243) that should resolve this problem.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 14, 2018, 10:42am UTC](https://discuss.elastic.co/t/ml-job-hard-limit-error-on-linux/150865/12 "2018-11-14T10:42:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.