# Machine learning can't detect changed pattern in log rate

**URL:** <https://discuss.elastic.co/t/machine-learning-cant-detect-changed-pattern-in-log-rate/305163>\
**Category:** Kibana\
**Tags:** elastic-stack-machine-learning\
**Created:** [May 19, 2022, 10:04am UTC](https://discuss.elastic.co/t/machine-learning-cant-detect-changed-pattern-in-log-rate/305163 "2022-05-19T10:04:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\
**Post date:** [May 19, 2022, 10:04am UTC](https://discuss.elastic.co/t/machine-learning-cant-detect-changed-pattern-in-log-rate/305163/1 "2022-05-19T10:04:23Z")

</div>

Hello, I created a machine learning job using the API, I'm interested in high and low log rate for kubernetes containers, and we are trying to detect those unusual low/high log rate when they occur.

Here is the job ML definition:

```auto
PUT _ml/anomaly_detectors/foo-ml-monitor-logs
{
  "description": "Monitor unusual log rate, missing or high log rate",
  "groups": [
    "foo"
  ],
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "function": "high_count",
        "partition_field_name": "kubernetes.container.name.keyword",
        "detector_description": "high_count on partition field kubernetes.container.name.keyword"
      },
      {
        "function": "low_count",
        "partition_field_name": "kubernetes.container.name.keyword",
        "detector_description": "low_count partition_field_name=\"kubernetes.container.name.keyword\""
      }
    ],
    "influencers": []
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "model_plot_config": {
    "enabled": false,
    "annotations_enabled": true
  },
  "results_index_name": "foo-ml-monitor-logs",
  "analysis_limits": {
    "model_memory_limit": "13MB"
  }
}

```

after processing documents, here is the result of a container tagged with high severity:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/9/598def69b30db1b1be0e1944d9e246e62e1e1ab3.png)

We can see that we have a seasonality, log rate is low during night but we are interested in the last part (circle in red) where the pattern change, i.e no more seasonality and indeed we faced a log interruption during this time. How can I update the model to catch such a behavior please? is it possible with ML?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 19, 2022, 5:04pm UTC](https://discuss.elastic.co/t/machine-learning-cant-detect-changed-pattern-in-log-rate/305163/2 "2022-05-19T17:04:35Z")

</div>

There's not quite enough information to tell what's transpired here, but certainly, the ML job should (very easily) catch this situation. Since you have a `partition_field_name` defined, your screenshot graph will be for a particular `kubernetes.container.name.keyword`...so...

It would be good to have you create another test job, but just for that `kubernetes.container.name.keyword`, then also set `"enabled" : true` for `model_plot_config`. Then, run the job only from April 1, 2022 up through the problematic period - and see that it looks like.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 16, 2022, 5:05pm UTC](https://discuss.elastic.co/t/machine-learning-cant-detect-changed-pattern-in-log-rate/305163/3 "2022-06-16T17:05:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.