The ML team recently introduced transforms which is very much geared towards behavioural analytics. It allows you to summarise behaviour of each entity using the aggregations framework and you can then run analysis on the "entity-centric" indices it creates.
But rather than the count of 'action A' and 'action B', I want to know how many times 'action A' and 'action B' occur in succession.
And when the number of consecutive occurrences is high, i want to detect this as an outlier.
For example, if a pattern is repeated in which a particular user attempts to connect to two specific ports continuously, we want to detect this as an outlier.
Sequences are harder to spot currently (it involves Painless scripting). Our recent acquisition of Endgame has provided technology to tackle this - the EQL language it provides is designed for these types of questions but we have work to do to incorporate into the core elasticsearch engine.
In the interim, you can write Painless scripts as parted of a ‘scripted’ aggregation using the transform api or write an old fashioned entity-centric update script.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
