# How to create outlier jobs with data fields coming from multiple sources (log1,log2, metricbeat1, etc....) **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [October 29, 2021, 4:10pm UTC](https://discuss.elastic.co/t/how-to-create-outlier-jobs-with-data-fields-coming-from-multiple-sources-log1-log2-metricbeat1-etc/288016 "2021-10-29T16:10:24Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Indigo\_Star](https://avatars.discourse-cdn.com/v4/letter/i/ba8739/32.png) [@Indigo\_Star](https://discuss.elastic.co/u/Indigo_Star)\ **Post date:** [October 29, 2021, 4:10pm UTC](https://discuss.elastic.co/t/how-to-create-outlier-jobs-with-data-fields-coming-from-multiple-sources-log1-log2-metricbeat1-etc/288016/1 "2021-10-29T16:10:24Z")
Hi everyone, Let me elaborate what i want and why i want that then may be it would be easier to find a best solution. I am new to ML jobs, I've created Single metric jobs and also have explored Multi-metric ML jobs a little bit. I know we can add influencers to the Multi-Metric jobs. Now i need to create a real scenario job. Where there can be multiple detectors and multiple influencers. We have microservices, all logs and metric beat data is coming from them into 1 Kibana index. I have extracted fields of interest This is an outlier detection ML job. So we are getting data points from different sources. And data fields are specific to a specific source but they may have an impact on each other. Following is a sample data set, all the fields are available in one index but some documents would have some of the data fields and others would have some others. index = 2021IDX proecessing\_time, slow\_query, cpu\_usage\_service1, cpu\_usage\_service2, data\_field2\_log1, data\_field1\_log2, data\_field1\_metricbeat1, data\_field1\_metricbeat2, database\_fragmentation\_count\_metricbeat2 My question is: Do i really need to combine those events somehow so all the data fields of interest are available on each event/doc? If not does it leave any impact on the ML job behavior? e.g. event 1 looks like this: 2021/1/1T12:0:0.162, data\_field1\_metricbeat2 , database\_fragmentation\_count\_metricbeat2, cpu\_usage\_service1 event 2 looks like this: 2021/1/1T12:0:0.180, proecessing\_time , data\_field1\_metricbeat1, data\_field2\_log1, slow\_query event 3 looks like this: 2021/1/1T12:0:1.178, cpu\_usage\_service2 , data\_field1\_log2 Now If i create an ML Job with these fields some as metric/detector some as influencer. How the ELK would combine all these fields to consider them as one row? I know jobs does aggregation of events over the given time bucket but to keep things simple i am just talking about 1 instance of each type of event. To address this i was thinking to combine them all into one. Is it really required in my scenario? Or i don't have to worry about it and ML job would take care? I know for sure that Data frame analytics is not working for this scenario, When i create the data frame analytics it only shows some fields to include in the job. And to include others i need to filter the data based by the 'log-type' only then i can see those fields but in that case the fields from other logs are not available. Considering this I still feel that we need to have them available in all indexed documents, what do you say? I was thinking to combine them, but not sure if that's correct approach? I would really appreciate if i can get any guidance on this. I don't find these scenarios explained anywhere in the documentation or videos. Thanks ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [November 1, 2021, 9:07pm UTC](https://discuss.elastic.co/t/how-to-create-outlier-jobs-with-data-fields-coming-from-multiple-sources-log1-log2-metricbeat1-etc/288016/2 "2021-11-01T21:07:24Z")
This is being discussed in a separate thread here: [How to aggregate multiple events coming from different logs with slight different timestamp, when the only field is timestamp to combine those? - #4 by richcollier](https://discuss.elastic.co/t/how-to-aggregate-multiple-events-coming-from-different-logs-with-slight-different-timestamp-when-the-only-field-is-timestamp-to-combine-those/287848/4) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 29, 2021, 9:08pm UTC](https://discuss.elastic.co/t/how-to-create-outlier-jobs-with-data-fields-coming-from-multiple-sources-log1-log2-metricbeat1-etc/288016/3 "2021-11-29T21:08:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.