I am looking for help/advice in regards to using the out of the box ML jobs. Specifically, some of the jobs that are underlying SIEM detections. I have an environment with 4500 hosts sending endpoint data and some network data to the stack. The analysts (who are the customer) have access to the Elastic stack and an Endgame appliance. Right now they are using Endgame pretty heavily and use the logs we are ingesting to correlate some of the things they are seeing in Endgame. I am hoping to use ML and the security app to augment what is already in Endgame.
The ideal situation is that they will alter their workflow to use the Elastic Security App and ML much more than they do now. The hurdle is Endgame is really good, and I am having a tough time figuring out what I can do in Elastic that is a genuine value add to Endgame.
The issue that I am running into is, the out of the box security detections in Elastic are a little redundant to what is being seen in Endgame. I am trying to figure out how to extract more value out of the Security features within Elastic (if possible).
I have spent some time running about 10 of the out of the box ML jobs. These jobs are doing what they are supposed to do from a ML perspective and are identifying anomalies. The issue is the environment is pretty big, complex and diverse so unique things happen frequently.
For instance rare_process_by_host_windows_ecs. It is detecting a lot of benign processes being run by admin accounts, service accounts or regular users. I could just tune these accounts out or make a running list of those processes, but doing both of those things conflict with my idea of ML. If there was a way to pull in the hashes of these files and then throw those against a database with malicious hashes in it then we would be in business. We are working on this solution, but to be honest we could just run an indicator match query once we ingest the threat intel data w/o bothering w/ ML.
The next example is v2_windows_anomalous_user_name_ecs. Here again there are quite a few account types that we can get rid of, but there are still a huge number of anomalies over a 7 day period, something like 400 once I get rid of all service accounts and admins (again pretty high value accounts).
I will try to get to the point. Have you guys had any success with the out of the box jobs or have you had to build your own jobs? I don't have a problem building my own jobs, and the out of the box jobs have given me a good starting place and a chance to learn a little bit more about elastic ML, but I don't want to sink to much time into an endeavor that might not pay off.