How to get XPACK running for Detecting DNS Data Exfiltration

[octogar]

On this page: https://www.elastic.co/what-is/elasticsearch-machine-learning/recipes/dns-data-exfiltration-tunneling

..when you click on the example there are directions: https://github.com/elastic/examples/blob/master/Machine%20Learning/Security%20Analytics%20Recipes/dns_data_exfiltration/EXAMPLE.md

..but these directions are outdated, I FINALLY figured out that x-pack is automatically installed in recent versions of elasticsearch, BUT I have been having trouble with the obviously dated instructions on this elasticsearch page... Do I need a license to run x-pack to be able to do this DNS Data Exfiltration machine learning? And if so, what license do I need??

https://www.elastic.co/subscriptions is not helpful.

[sophie_chang]

Hi

I have raised this issue regarding the supported versions of the recipes. https://github.com/elastic/examples/issues/264

• The X-Pack features are included in the default distribution since 6.3.
• Machine learning features are available with a trial or platinum license.

At lot has changed since v5.4 as I am sure you are experiencing. Looking at the ML job and datafeed config, I can see some paramaters which have since been deprecated or removed from the example job and datafeed json configs. When creating the ML job I would recommend using the Kibana UI, rather than the scripts provided.

In the UI:

• Create Job
• Select the packetbeat index (this should have been indexed already, and a kibana index pattern created)
• Select Advanced Job Wizard
• Create the job with the following config

Detector: high_info_content(sub_domain) over domain exclude_frequent=all
Bucketspan: 5m
Influencer(s): client_ip, beat.name, domain

If you do not already have some familiarity with creating ML jobs, then I would recommend looking at the Single Metric Wizard first, and analyzing a simple event rate count of a data source.

Best wishes
Sophie

[willemdh]

It would be nice if the example ml recipes would use real-live current ecs compatible field names..

[octogar]

Thank you Sophie, hopefully someone who knows how to get this running can help us out.

[richcollier]

To answer your question from earlier - you do need a Platinum license to have ML run this kind of DNS exfiltration detection job (or just a trial license if you want to test it short term).

[octogar]

Thank you, Rich. Sophie did have this in her answer.

In reply to your comment at ML recipes require updating for recent stack versions · Issue #264 · elastic/examples · GitHub ...

I personally think that the recipes should be removed from the website and instead the "recipes" should be just whatever gets documented as the ML jobs for the SIEM app.

I just want to gently nudge that if there were updated directions that show how it is possible to get machine learning running to detect DNS exfiltration on the recent versions of Elasticsearch, that you will probably get a lot more subscriptions.

Not everyone is using or even planning to set up the SIEM app yet.

But just in case I end up setting up the SIEM app much later, are the already directions documented somewhere for ML in the SIEM app? Or no directions for neither Elasticsearch nor SIEM app?

[richcollier]

Hi, thanks for your input.

FYI - the ML jobs for SIEM are publicly documented and available for everyone to see, even if you're not using the SIEM app. And, the actual job configurations are right on GitHub - and these configurations will always be up to date, and versioned. For example, here is the configuration of the DNS exfil job that the SIEM app uses. All SIEM app ML jobs use the ECS-compliant field names.

For this reason, this is why I feel the removal of the outdated recipes on elastic.co makes sense.

[cwurm (Christoph Wurm)]

Just to clarify:

1. The DNS exfil job is scheduled to be added (together with other Packetbeat jobs) to the SIEM app in the upcoming 7.5 release. 16 other jobs for Auditbeat and Winlogbeat are already included today in 7.4.
2. If you use the official hosted Elasticsearch Service (Elastic Cloud) then Machine Learning is included even for small, inexpensive deployments.