Well done to ElasticSearch on the "Machine-Learning" modules. Just ran some data through to get a feel and love it.
Great Job
7 Likes
My thoughts exactly... actually I used the words "bad ass" when sharing my initial impressions! 
4 Likes
I anyone is interested in my initial observations I have shared them in a Linkedin article...
https://www.linkedin.com/pulse/first-look-elastics-new-machine-learning-technology-robert-cowart
8 Likes
Hi, Did you have any issues when setting it up initially? I get errors about it not being able to read the jobs and in the logs I see a 500 Error. Did you install from scratch or upgrade from a previous release? Do you have security enabled?
Steve
1 Like
I didn't have any issues at all, although I do NOT have security enabled. This was a 5.3.0 environment that I first upgraded to 5.4.0, and made sure everything was working. I then installed X-Pack, disabling security before restarting after the X-Pack install.
I have 4 or 5 environments running at any given time for solution development purposes, if I learn anything related to your issues as I apply to other environments, I will let you know.
Rob
1 Like
Does it support alert notifications for anomalies like in watcher?
1 Like
Yes! The anomaly results are written into an ES index, so you can directly use Watcher on the results today. Going forward, we have plans to more explicitly enable alerting/notifications right from the Machine Learning UI and the Watcher UI, the first iteration of which was also introduced in 5.4.
2 Likes
Just a stupid question, is the ml-beta only usable in the platinum subscription, or can one test it another way?
1 Like
You can see all the ML features for a limited time when using a trial license.
1 Like
Indeed - when you first install X-Pack, it comes with a built-in 30 day trial license that enables all functionality. When that expires, you can either register for a free basic license, which enables only some functionality, or you can reach out to our team at sales@elastic.co, and we can quickly issue you an extended Trial license.
If you have a Basic license already, and you'd like to work with Machine Learning, just send a note to sales@elastic.co requesting a trial license extension and we'll send one out.
Rob, How did you do the URL's?
If you wouldn't mind elaborating on that.
For the Custom URL I found the prelert docs work
Configuring custom URLs
Custom URLs provide links from the “Anomalies” table in the Explorer window to custom dashboards or external websites, allowing the end user for example to drill into the source data at the time of an anomaly.
Multiple custom URLs can be defined in the Job Details tab at job creation and are stored in the job configuration. Custom URLs can also be created or edited for an existing job. For each custom URL, two properties are configured: a label which is used as the text in the links menu for an anomaly in the Explorer window, and the URL of the link itself.
String substitution in custom URLs
Dollar sign ($) delimited tokens can be used in a custom URL which will then be substituted for the values of the corresponding fields from the anomaly records stored in Elasticsearch. For example, for a configured URL of http://cassandra.datastore.com/dashboards?user=$user_name$, the value of the user_name field from the anomaly record will be substituted into the $user_name$ token when clicking on the link in the Explorer “Anomalies” table.
Four keywords can be used as tokens for String substitution in a custom URL which play a special role when the link is opened: $earliest$, $latest$, $prelertcategoryterms$ and $prelertcategoryregex$.
$earliest$ and $latest$ in custom URLs
$earliest$ and $latest$ tokens are used to pass the time span of the selected anomaly to the target page. The tokens will be substituted with date-time Strings in ISO-8601 format, e.g. 2016-02-08T00:00:00.000Z, as used by Kibana for example when displaying times in dashboards.
When clicking on the custom URL from the Explorer window, if the data in the “Anomalies” table is aggregated by hour, then one hour either side of the anomaly time will be used for the earliest and latest times. If aggregated by day, then the start and times of that day will be used i.e. from 00:00:00.000 to 23:59:59.999. If the Anomalies table is set to “Show All” with no aggregation of anomalies, then the start and end times of the anomaly bucket will be used.
$prelertcategoryterms$ and $prelertcategoryregex$ in custom URLs
For jobs which are analyzing data based on the categorization of values, $prelertcategoryterms$ and $prelertcategoryregex$ can be used to pass on details of the category definition for the selected anomaly to the target page. If present in the custom URL, the tokens will be replaced with the category definition terms or regex for the category ID of the selected anomaly, using the value of the prelertcategory field from the anomaly record stored in Elasticsearch. For example, the following custom URL uses a $prelertcategoryterms$ token to open up a dashboard in a user’s Kibana installation to display source data stored in an it_ops_app index for a job using a categorization detector:
http://localhost:5601/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:'$earliest$',mode:quick,to:'$latest$'))&_a=(columns:!(_source),index:it_ops_app,interval:auto,query:(query_string:(analyze_wildcard:!t,query:'$prelertcategoryterms$')),sort:!('@timestamp',desc))
When substituting $prelertcategoryterms$, each of the terms are prefixed with a + character to ensure that the Elasticsearch Query String query run in a drilldown Kibana dashboard searches for all of the terms. Therefore if drilling into a non-Kibana URL, the target page should reformat the terms value to remove the + characters as necessary.
Notes on configuring custom URLs
Please be aware of the following points when configuring custom URLs for a job:
When creating a link to a Kibana dashboard, note that the URLs of dashboards can be very long so be careful of typos, end of line characters and URL encoding.
If an influencer name is used for string substitution e.g. $clientip$, it may not always be available in the results records. The link will still take you to the dashboard, however the query will remain as $clientip$ and will need to be manually corrected.
Dates substituted for $earliest$ and $latest$ tokens will be in ISO-8601 format and the target system needs to understand this.
If the job performed an analysis against nested JSON fields, the tokens for String substitution may refer to these fields using dot notation e.g. $cpu.total$.
Elasticsearch source data mappings may make it difficult for the query string to work. Test the custom URL before saving the job configuration to check it will work as expected, particularly when using String substitution.
Yeah, that is what I used as a starting point. I understand that the Prelert docs will be all reworked and updated, but for now the needed info is spread between the two sites. I assume this is part of the reason that it is all still "beta".
I received multiple requests for additional info on the URL linking. That seems to be a much desired use-case. As Elastic continues adding more and more "wizards" and "builders" into the toolset, it would be great to see a URL builder for ML.
Rob
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.