# ML Multi-Metric query fails when similar Single-Metric is OK

**URL:** <https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [August 24, 2017, 1:15pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240 "2017-08-24T13:15:58Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\
**Post date:** [August 24, 2017, 1:15pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/1 "2017-08-24T13:15:58Z")

</div>

I'm starting to play with ML jobs, but encountered an issue with Multi-Metric job.  
It fails with

Datafeed is encountering errors extracting data: [ml-multi-low-count-test]  
Search request returned shard failures; first failure: shard [[YuBCwg][logstash-general-2017.08.09][0]], reason [RemoteTransportException[[elasticsearch][127.0.0.1:9300]  
[indices:data/read/search[phase/query]]]; nested: QueryShardException[No mapping found for [@timestamp] in order to sort on]; ]; see logs for more info

Single-Metric job with the same indexes works nice. But as far as I see, time\_field is the same in both cases:

"data\_description": {  
"time\_field": "@timestamp",  
"time\_format": "epoch\_ms"

Any ideas how to debug this?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 25, 2017, 7:37pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/2 "2017-08-25T19:37:38Z")

</div>

Hi Vitaly,

Please do the following for both jobs (the working single-metric and the non-working multi-metric)

```auto
curl -u elastic:changeme -XGET 'localhost:9200/_xpack/ml/anomaly_detectors/<job_id>?pretty'

```

```auto
curl -u elastic:changeme -XGET 'localhost:9200/_xpack/ml/datafeeds/datafeed-<job_id>?pretty'

```

Then we can compare the two configurations

---

<div class="post-metadata">

**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\
**Post date:** [August 27, 2017, 5:13pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/3 "2017-08-27T17:13:18Z")

</div>

Thank you, please see below:

1. 

{  
"count" : 1,  
"jobs" : [  
{  
"job\_id" : "ml-test-single-metric-low-count",  
"job\_type" : "anomaly\_detector",  
"job\_version" : "5.5.2",  
"description" : "2nd 24.08.2017",  
"create\_time" : 1503565565772,  
"finished\_time" : 1503565567428,  
"analysis\_config" : {  
"bucket\_span" : "5m",  
"summary\_count\_field\_name" : "doc\_count",  
"detectors" : [  
{  
"detector\_description" : "low\_count",  
"function" : "low\_count",  
"detector\_rules" : [],  
"detector\_index" : 0  
}  
],  
"influencers" : []  
},  
"data\_description" : {  
"time\_field" : "@timestamp",  
"time\_format" : "epoch\_ms"  
},  
"model\_plot\_config" : {  
"enabled" : true  
},  
"model\_snapshot\_retention\_days" : 1,  
"model\_snapshot\_id" : "1503828750",  
"results\_index\_name" : "shared"  
}  
]  
}  
2)  
{  
"count" : 1,  
"jobs" : [  
{  
"job\_id" : "ml-multi-low-count-test",  
"job\_type" : "anomaly\_detector",  
"job\_version" : "5.5.2",  
"description" : "3rd",  
"create\_time" : 1503568129335,  
"finished\_time" : 1503568130891,  
"analysis\_config" : {  
"bucket\_span" : "5m",  
"detectors" : [  
{  
"detector\_description" : "low\_count",  
"function" : "low\_count",  
"partition\_field\_name" : "type.keyword",  
"detector\_rules" : [],  
"detector\_index" : 0  
}  
],  
"influencers" : [  
"type.keyword"  
]  
},  
"data\_description" : {  
"time\_field" : "@timestamp",  
"time\_format" : "epoch\_ms"  
},  
"model\_snapshot\_retention\_days" : 1,  
"results\_index\_name" : "shared"  
}  
]  
}

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 28, 2017, 1:44pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/4 "2017-08-28T13:44:08Z")

</div>

> [@richcollier](#):
>
> curl -u elastic:changeme -XGET 'localhost:9200/\_xpack/ml/datafeeds/datafeed-\<job\_id\>?pretty'

Vitaly - Please also collect the output from the data feed config for each job. See above.

---

<div class="post-metadata">

**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\
**Post date:** [August 28, 2017, 4:29pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/5 "2017-08-28T16:29:46Z")

</div>

Thank you, here it is:

{  
"count" : 1,  
"datafeeds" : [  
{  
"datafeed\_id" : "datafeed-ml-multi-low-count-test",  
"job\_id" : "ml-multi-low-count-test",  
"query\_delay" : "60s",  
"frequency" : "150s",  
"indices" : [  
"logstash-\*"  
],  
"types" : [  
"newrelic-nrsysmond",  
"bvpyzabbix",  
"rabbitmq",  
........  
],  
"query" : {  
"match\_all" : {  
"boost" : 1.0  
}  
},  
"scroll\_size" : 1000,  
"chunking\_config" : {  
"mode" : "auto"  
}  
}  
]  
}

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 28, 2017, 5:29pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/6 "2017-08-28T17:29:17Z")

</div>

> [@richcollier](#):
>
> Vitaly - Please also collect the output from the data feed config for each job. See above.

^^ missing the one for the single metric job

My guess is that it's possible that in the multi-metric job, there are multiple data "types" in your "logstash-\*" which have a different timestamp field from each other?

Just so you know - "types" are being depcreated by Elasticsearch in v6.0. Just so you're prepared...

---

<div class="post-metadata">

**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\
**Post date:** [August 28, 2017, 7:10pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/7 "2017-08-28T19:10:59Z")

</div>

> Just so you know - “types” are being depcreated by Elasticsearch in v6.0.  
> Just so you’re prepared…  
> Thank you 🙂  
> BTW, there are about 20 types, I just cutted out.

it was in my previous letter, after separator.  
here it is:

{  
"count" : 1,  
"datafeeds" : [  
{  
"datafeed\_id" : "datafeed-ml-test-single-metric-low-count",  
"job\_id" : "ml-test-single-metric-low-count",  
"query\_delay" : "60s",  
"frequency" : "150s",  
"indices" : [  
"logstash-\*"  
],  
"types" : [  
"newrelic-nrsysmond",  
.........  
],  
"query" : {  
"match\_all" : {  
"boost" : 1.0  
}  
},  
"aggregations" : {  
"buckets" : {  
"date\_histogram" : {  
"field" : "@timestamp",  
"interval" : 300000,  
"offset" : 0,  
"order" : {  
"\_key" : "asc"  
},  
"keyed" : false,  
"min\_doc\_count" : 0  
},  
"aggregations" : {  
"@timestamp" : {  
"max" : {  
"field" : "@timestamp"  
}  
}  
}  
}  
},  
"scroll\_size" : 1000,  
"chunking\_config" : {  
"mode" : "manual",  
"time\_span" : "300000000ms"  
}  
}  
]  
}

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 28, 2017, 7:15pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/8 "2017-08-28T19:15:25Z")

</div>

> [@Vitaly\_il](#):
>
> “bvpyzabbix”,  
> “rabbitmq”,

Right so the types above ^^ are extraneous compared to the config of the single-metric job (which works for you). Clone your existing multi-metric job (to keep most of the config parameters) but remove the above extraneous types, then try to run the job....

---

<div class="post-metadata">

**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\
**Post date:** [August 29, 2017, 4:41am UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/9 "2017-08-29T04:41:37Z")

</div>

Sorry for confusing output - "types" in both cases are the same. I just  
reduced output in different way (in original output there are 50 types)

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [August 29, 2017, 11:50am UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/10 "2017-08-29T11:50:08Z")

</div>

Oh ok! That's good to know.

I think that you must have one (or more) of those 50 types with a missing mapping for `@timestamp` (??)

In the single-metric job, the query to the index automatically includes a `date_histogram` aggregation on the field `@timestamp` (as you can see above). The multi-metric job does not do this. So, perhaps, the single-metric job's aggregation is masking the problem in your data?

I can also suggest that you inspect the elasticsearch.log file when hitting the datafeed "preview" for the problematic job:

`GET _xpack/ml/datafeeds/datafeed-ml-multi-low-count-test/_preview/`

---

<div class="post-metadata">

**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\
**Post date:** [August 30, 2017, 1:16pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/11 "2017-08-30T13:16:07Z")

</div>

Thank you, it really was one of the indices without proper timestamp mapping. Job is running as far as we don't include this index.  
Vitaly

---

<div class="post-metadata">

**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\
**Post date:** [September 7, 2017, 8:33am UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/12 "2017-09-07T08:33:52Z")

</div>

Rich,

I'll appreciate your help again. After further investigation it seems that ML job fails on indices without proper @timestamp filed mapping, but on empty indices.

Job fails with this message:

Datafeed is encountering errors extracting data: [ml-multicount-all-indices-count-test] Search request returned shard failures; first failure: shard [[pREMVKEzTWe-J3vLYuBCwg][logstash-general-2017.09.02][0]], reason [RemoteTransportException[[staging-elk-elasticsearch-][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: QueryShardException[No mapping found for [@timestamp] in order to sort on]; ]; see logs for more info

This index is empty, and mapping is:

"logstash-general-2017.09.02": {  
"mappings": {  
"_default_": {  
"properties": {  
"@timestamp": {  
"type": "date"  
},  
"geoip": {  
"properties": {  
"location": {  
"type": "geo\_point"  
}  
}  
}  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [September 8, 2017, 6:44pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/13 "2017-09-08T18:44:15Z")

</div>

Vitaly,

Under what conditions do you have empty indices? That shouldn't be the case if you're using daily indices.

Also, I'll let you know again that best practices, you should separate different kinds of data to different indices. In v6.0, Elastic will be [deprecating the "\_type" mechanism](https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html), which allows different types of data to exist in the same physical index.

---

<div class="post-metadata">

**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\
**Post date:** [September 10, 2017, 6:01pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/14 "2017-09-10T18:01:02Z")

</div>

> [@richcollier](#):
>
> Under what conditions do you have empty indices? That shouldn't be the case if you're using daily indices.

As far as I see, logstash output filter send wrong formatted data - like two comma-separated strings in "type" field.

> [@richcollier](#):
>
> Also, I'll let you know again that best practices, you should separate different kinds of data to different indices. In v6.0, Elastic will be deprecating the "\_type" mechanism, which allows different types of data to exist in the same physical index.

Yes, I remember. But AFAIK Elastic performance may be affected from a large number of shards?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 8, 2017, 6:01pm UTC](https://discuss.elastic.co/t/ml-multi-metric-query-fails-when-similar-single-metric-is-ok/98240/15 "2017-10-08T18:01:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.