How to create Machine Learning job using two indices from kibana UI Version 7.8.0

[prabhu_c]

Hi,
I have to create advanced machine learning job using Kibana 7.3 by pulling data from two indices.
I am creating job using Kibana UI. I am not able to understand how to pull data from two indices? while creating ML job we can only select one indices and not sure how to include other indices. Kindly help me on the steps to be followed.
note : I am using ML -kibana 7.8.0
Thank you
Prabhu

[richcollier]

Your question needs more detail. Do you mean "two indices" in the way that having an index pattern (such as metricbeat-*) matches more than one index? Or do you mean that you wish to "join" data that is different in types (fields)? Would having two jobs (one for each index) not be satisfactory?

[prabhu_c]

Hi Richcollier,
I was trying to do first one. i mean two indices having index pattern say metricbeat-* and nginx*. using these two indices i want to create one advanced machine learning job which will take one parameter is nginx say status code and i want to use other variable in metricbeat( cpu usage, memory usage) as influencer and create machine learning job.
The problem i am facing is , there is option to use only one index in kibana UI and not sure how to use other index, i tried changing json config and include other index, but its not working. appreciate your inputs on this.

Thank you
Prabhu

[richcollier]

If the data sets between the two indices are different types (i.e. nginx logs and system metrics) you will need two separate jobs.

You can combine the results of those jobs either visually (with the Anomaly Explorer UI) or programmatically via an alert (Watch) if there was a desire to know any correlative behavior between the jobs. There is also the overall_buckets API call: https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ml-get-overall-buckets.html

[prabhu_c]

Hi Richcollier,
Many thanks for your inputs. does that mean we can create machine learning job using only one index? I was trying to use two index( nginx and metricbeat) and create one advanced machine learning job.
essentially i heard that , we can construct datafeed of having multiple indices and then create machine learning job and infer results from one job instead of multiple jobs.
could you confirm whether this can be done in current kibana version 7.8.0.
Thank you,
Prabhu

[richcollier]

ML jobs (specifically the datafeed) can only reference one "index pattern". So, technically, the index pattern can span multiple indices. However, it is not recommended to have the ML job try to process different "kinds" of data in the same job. You can get mapping clashes and the datafeed will error out.

So, the recommended approach is to create an ML job for every distinct data "type" (i.e. one for Nginx logs, one for metricbeat data, etc.).

[prabhu_c]

Hi Richcollier,
Many thanks for your input and suggested approach.
As you mentioned in your previous post "You can combine the results of those jobs either visually (with the Anomaly Explorer UI) or programmatically via an alert (Watch) if there was a desire to know any correlative behavior between the jobs. "

If i create two jobs for each index pattern, can you please let us know steps to combine visually in anomaly explorer UI and programmatically via alert( as you mentioned i want to study if any correlation exists between two)
Also, just wanted to know can we pull results of both the jobs to one dashboard in kibana?
Many thanks for your inputs and appreciate your help.

Thank you
Prabhu

[richcollier]

In Anomaly Explorer, simply choose more than one job to view by clicking the "Edit job selection" link near the top left, and select the jobs you want to overlay:

The end result will look something like:

Notice the "vertical stripe" of correlated red and yellow anomalies across all four jobs about 3/4ths of the way through the time line.

To combine the results of multiple jobs in an alert, you can either leverage the overall_buckets API call: https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ml-get-overall-buckets.html or you could use a chained input watch to query the results of multiple ML jobs. An example of that is described in my book and the watch definition is also provided.

[prabhu_c]

Many thanks for your valuable inputs. appreciate your response. Thank you.