# How to create Machine Learning job using two indices from kibana UI Version 7.8.0 **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-machine-learning\ **Created:** [June 29, 2020, 3:54am UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049 "2020-06-29T03:54:36Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![prabhu\_c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhu_c/32/57387_2.png) [@prabhu\_c](https://discuss.elastic.co/u/prabhu_c)\ **Post date:** [June 29, 2020, 3:54am UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/1 "2020-06-29T03:54:36Z")
Hi, I have to create advanced machine learning job using Kibana 7.3 by pulling data from two indices. I am creating job using Kibana UI. I am not able to understand how to pull data from two indices? while creating ML job we can only select one indices and not sure how to include other indices. Kindly help me on the steps to be followed. note : I am using ML -kibana 7.8.0 Thank you Prabhu ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [June 30, 2020, 2:46pm UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/2 "2020-06-30T14:46:35Z")
Your question needs more detail. Do you mean "two indices" in the way that having an index pattern (such as `metricbeat-*`) matches more than one index? Or do you mean that you wish to "join" data that is different in types (fields)? Would having two jobs (one for each index) not be satisfactory? ---
**Author:** ![prabhu\_c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhu_c/32/57387_2.png) [@prabhu\_c](https://discuss.elastic.co/u/prabhu_c)\ **Post date:** [July 1, 2020, 3:53am UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/3 "2020-07-01T03:53:50Z")
Hi Richcollier, I was trying to do first one. i mean two indices having index pattern say metricbeat-\* and nginx\*. using these two indices i want to create one advanced machine learning job which will take one parameter is nginx say status code and i want to use other variable in metricbeat( cpu usage, memory usage) as influencer and create machine learning job. The problem i am facing is , there is option to use only one index in kibana UI and not sure how to use other index, i tried changing json config and include other index, but its not working. appreciate your inputs on this. Thank you Prabhu ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [July 1, 2020, 12:20pm UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/4 "2020-07-01T12:20:20Z")
If the data sets between the two indices are different types (i.e. nginx logs and system metrics) you will need two separate jobs. You can combine the results of those jobs either visually (with the Anomaly Explorer UI) or programmatically via an alert (Watch) if there was a desire to know any correlative behavior between the jobs. There is also the `overall_buckets` API call: [https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ml-get-overall-buckets.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ml-get-overall-buckets.html) ---
**Author:** ![prabhu\_c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhu_c/32/57387_2.png) [@prabhu\_c](https://discuss.elastic.co/u/prabhu_c)\ **Post date:** [July 1, 2020, 1:43pm UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/5 "2020-07-01T13:43:15Z")
Hi Richcollier, Many thanks for your inputs. does that mean we can create machine learning job using only one index? I was trying to use two index( nginx and metricbeat) and create one advanced machine learning job. essentially i heard that , we can construct datafeed of having multiple indices and then create machine learning job and infer results from one job instead of multiple jobs. could you confirm whether this can be done in current kibana version 7.8.0. Thank you, Prabhu ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [July 1, 2020, 3:05pm UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/6 "2020-07-01T15:05:03Z")
ML jobs (specifically the datafeed) can only reference one "[index pattern](https://www.elastic.co/guide/en/kibana/current/tutorial-define-index.html)". So, technically, the index pattern can span multiple indices. However, it is _not recommended_ to have the ML job try to process different "kinds" of data in the same job. You can get mapping clashes and the datafeed will error out. So, the recommended approach is to create an ML job for every distinct data "type" (i.e. one for Nginx logs, one for metricbeat data, etc.). ---
**Author:** ![prabhu\_c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhu_c/32/57387_2.png) [@prabhu\_c](https://discuss.elastic.co/u/prabhu_c)\ **Post date:** [July 2, 2020, 3:26am UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/7 "2020-07-02T03:26:24Z")
Hi Richcollier, Many thanks for your input and suggested approach. As you mentioned in your previous post "You can combine the results of those jobs either visually (with the Anomaly Explorer UI) or programmatically via an alert (Watch) if there was a desire to know any correlative behavior between the jobs. " If i create two jobs for each index pattern, can you please let us know steps to combine visually in anomaly explorer UI and programmatically via alert( as you mentioned i want to study if any correlation exists between two) Also, just wanted to know can we pull results of both the jobs to one dashboard in kibana? Many thanks for your inputs and appreciate your help. Thank you Prabhu ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [July 3, 2020, 12:54pm UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/8 "2020-07-03T12:54:27Z")
In Anomaly Explorer, simply choose more than one job to view by clicking the "Edit job selection" link near the top left, and select the jobs you want to overlay: ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/2/325b90b435b9ff8bfd03b5a4f9c825104138ec34.png) The end result will look something like: ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/f/3fa6480ef9400d26dc5e6c79f5fc0e7e16405a1c.png) Notice the "vertical stripe" of correlated red and yellow anomalies across all four jobs about 3/4ths of the way through the time line. To combine the results of multiple jobs in an alert, you can either leverage the `overall_buckets` API call: [https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ml-get-overall-buckets.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ml-get-overall-buckets.html) or you could use a [chained input watch](https://www.elastic.co/guide/en/elasticsearch/reference/current/input-chain.html) to query the results of multiple ML jobs. An example of that is described in [my book](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-elastic-stack) and the [watch definition](https://github.com/PacktPublishing/Machine-Learning-with-the-Elastic-Stack/blob/master/Chapter06/custom_ML_watch.json) is also provided. ---
**Author:** ![prabhu\_c](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabhu_c/32/57387_2.png) [@prabhu\_c](https://discuss.elastic.co/u/prabhu_c)\ **Post date:** [July 6, 2020, 3:49am UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/9 "2020-07-06T03:49:05Z")
Many thanks for your valuable inputs. appreciate your response. Thank you. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 3, 2020, 3:49am UTC](https://discuss.elastic.co/t/how-to-create-machine-learning-job-using-two-indices-from-kibana-ui-version-7-8-0/239049/10 "2020-08-03T03:49:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.