Is ML capable to detect RAM saturation?

Elastic Stack Elasticsearch
1

Hi,

I test with ML a situation where RAM usage keeps increasing linearly and eats almost all RAM capacity.
The Machine Learning constructs the appropriate model but doesn't consider the fact that RAM usage is so close to saturation an anomaly.

Is there anything to do to let ML consider this type of case ?

2

If RAM usage is always increasing linearly, that is the normal behaviour and it reaching a specific level is not really an anomaly. If you have a fixed limit, that sounds like a use case for Watcher rather than ML.

3

Do you mean setting a static threshold in Watcher (ex: send alert if RAM usage > 3 go) ?

4

Not sure exactly what the condition would look like, but it does not necessarily need to be static.

5

Could you please explain a little bit more how ?

6

What does your data look like? Can you describe the condition you are looking to alert on?

7

I have gathered host metrics from my test machine using MetricBeat.
I applied ML job on a metric called system.memory.used.bytes (which is the RAM usage in bytes).
Now as a hypothesis, let's say that the RAM usage keeps growing linearly over time. I want to know if ML will realize that the system RAM has a limit, and therefore consider an anomaly the fact that RAM usage is nearly hits that limit.

8

ML cannot possibly automatically know that your linearly-growing metric has a practical limit. ML only knows the data's behavior in a naive way - meaning, it only knows things about data with respect to observations it has seen in the data's past.

However, to Christian's point - if you know there's an actual limit, then your use-case may be better suited as a static threshold in a Watch rather an ML job.

9

Clear answer.
Thank you all for your responses.

1 Like
10

You could use ML and the max function on system.memory.actual.used.pct which represents the percentage of RAM used in the system.
If the value increases a lot it’ll trigger an anomaly but technically if the value change from 1% of RAM used to 10% that would generate a pretty high anomaly score even if you still have a lot of RAM available.

It’d probably used watcher for your requirement.