Data mining firewall logs with machine learning

[mariofcs (Mario Chancay)]

Can anybody tell if the method described in the following url can be done with Elastic - Machine Learning :

Data mining firewall logs : Principal Component Analysis

Regards

Mario

[stevedodson]

We can perform a similar analysis using the 'population' analysis feature that is built into Elastic ML (https://www.elastic.co/guide/en/x-pack/current/ml-api-definitions.html#ml-detectorconfig - over_field_name). Population analysis automatically profiles entities across several dimensions and identifies unusual entities. Internally, this analysis uses techniques similar to PCA and will automatically cluster entities into peer groups.

As Elastic ML is native to the Elastic stack this can be simply run on data in Elasticsearch in real-time, and the methods we use are significantly less prone to relying on data spread, orthogonality and skew than naive PCA.

Given the interest in ML on firewall logs, we'll try to publish some suggested configurations and use cases shortly.