Event Correlation/Deviation

Elastic Stack Kibana
#1

I have a series of endpoints that have hourly readings as well as a set of event times. A specific subset of the endpoints' readings are expected to drop during the events. I am trying to figure out a visualization that could identify the endpoints that do not align with this expectation.

Right now the dataset I have includes only the device ID, reading, timestamp, and whether or not the device is in this subset. I am thinking about appending the data with a field that indicates whether or not that reading occurs during an event. While this would help identify the readings that occur during an event, I'm still not sure on how to identify individual devices that fail to deviate as expected.

Is this even possible and if so, what are some good strategies to approach this with?

#2

I'd be more inclined to use an ML job using population analysis, than a graph, if that's possible.

#3

@warkolm
I was considering that. Currently we're "proof-of-concept"ing this and so I am running the free version. I can activate the 30 day trial to test this but I want to be sure I have my basis covered before starting that. My basic understanding of ML is that it uses overall trends rather than event based correlations. Can the ML component correlate anomalies based on events rather than overall trends?

#4

ML has to ability to look at the values of fields or the rate of occurrence (the "count") of the number of results (documents) that match a query. You can do this either in a temporal sense (look at the values/rates over time and find the deviation with respect to an entity's own history) or compare entities against their peers in a homogeneous population (this is what @warkolm was mentioning).

So, if you want to locate entities that have less/more occurrences of things than their peers and consider them "outliers", then yes, a population analysis style of ML job might be right up your alley here.

Further reading: https://www.elastic.co/blog/temporal-vs-population-analysis-in-elastic-machine-learning

#5

You can also run a 14 day trial on our Elasticsearch Service, which include ML and more.

#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.