# Machine Learning broke somehow

**URL:** <https://discuss.elastic.co/t/machine-learning-broke-somehow/132100>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [May 16, 2018, 11:10am UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100 "2018-05-16T11:10:33Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 16, 2018, 11:10am UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/1 "2018-05-16T11:10:33Z")

</div>

Sorry about the topic, but I honestly have no clue what is the issue.

I was using Elastic stack 6.2.2, had four ML jobs and they were running fine.

Then I was adding a new job, and got this error immediately when I clicked save job:

> Save failed: [mapper\_parsing\_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current\_type [text], merged\_type [ObjectMapper]

Updated the whole stack from 6.2.2 to 6.2.4, same result, but now the old jobs didn't start to work.  
They all started and it said that the datafeed was running (also elasticsearch node logs said so), but there was an exclamation mark next to all the jobs, some said that no nodes found and some said something like "no model snapshot found".

I deleted all jobs and all .ml-\* indices, but same result:

> Save failed: [mapper\_parsing\_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current\_type [text], merged\_type [ObjectMapper]

Checked Elasticsearch log from all nodes, no logs regarding this issue.

I would appreciate if someone could help me to get the ML on its feet again 🙂

---

<div class="post-metadata">

**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\
**Post date:** [May 16, 2018, 12:24pm UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/2 "2018-05-16T12:24:41Z")

</div>

Hi admlko

When a job is created the mappings for the fields used by the job are added to the results index. In this case there must be a clash for the field `keywords` with an existing mapping. Note the index mappings are held in the cluster state so deleting the indices won't remove the mapping. Take a look a the [Mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-get-mapping.html)

The simplest fix is to use a dedicated index for the job's results then you won't get the mapping clash. When you create a job expand the Advanced section and check the Use dedicated index box as seen below.

![dedicated_index](https://us1.discourse-cdn.com/elastic/original/3X/0/2/026a38ca1aad24f533909d3c302934a3d2c39423.png)

> [@admlko](#):
>
> Save failed: [mapper\_parsing\_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current\_type [text], merged\_type [ObjectMapper]

Was this the only error you saw in the UI? I would have expected a more informative message.

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 16, 2018, 12:37pm UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/3 "2018-05-16T12:37:11Z")

</div>

Hi dkyle,

I am really glad you replied so fast!

> When a job is created the mappings for the fields used by the job are added to the results index. In this case there must be a clash for the field keywords with an existing mapping. Note the index mappings are held in the cluster state so deleting the indices won't remove the mapping. Take a look a the Mapping API

I have defined my own mappings for all indices I store my data. This particular mapping has a field named "keyword" which is stored both in text and keyword. This ML job uses "keywords.keyword".  
I have not changed the mapping in a long time, that's why I am a bit lost what has gone wrong.

> The simplest fix is to use a dedicated index for the job's results then you won't get the mapping clash. When you create a job expand the Advanced section and check the Use dedicated index box as seen below.

I **always** use dedicated indices for all ML jobs, exactly because of this. I have had issues before when some indices have the same field but different type.

> > Save failed: [mapper\_parsing\_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current\_type [text], merged\_type [ObjectMapper]  
> > Was this the only error you saw in the UI? I would have expected a more informative message.

Yes, this was the only error in UI. I was really surprised that there were no additional information in Elasticsearch node logs.

Thank you for looking into this 🙂

---

<div class="post-metadata">

**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\
**Post date:** [May 16, 2018, 2:26pm UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/4 "2018-05-16T14:26:00Z")

</div>

So you _are_ using a dedicated index?

Can you post the index mapping for the results index you are using and the data index. Use [GET mapping](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-get-mapping.html)

Because you can't save the job you probably won't have the JSON configuration but could you describe the job's analysis config and detector functions please. If you are creating an advanced job you can copy this from the edit json tab before saving the job.

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 16, 2018, 3:43pm UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/5 "2018-05-16T15:43:12Z")

</div>

> So you are using a dedicated index?

Yup 🙂

> Can you post the index mapping for the results index you are using and the data index. Use GET mapping

Cannot post the results index mapping because it doesn't create one.  
I just tried to create new job, [here's the JSON config](https://privatebin.net/?abb1f2e2f5d62e26#JcForRIOm2iwkZ7xtTIkz1EOdbyuA2KfxTxD4Z+vRC0=)

And when I click "Save":

> Save failed: [mapper\_parsing\_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current\_type [text], merged\_type [ObjectMapper]

I am expecting to see `.ml-anomalies-custom-adaudit-ml-test` index, but it is not created.

[Mapping for the data index](https://privatebin.net/?c765ff9c71416105#fkRzsT0Z0nyCU1K5kbTcfzNbkpPdwNf6Z+bAOROJ+YI=)

---

<div class="post-metadata">

**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\
**Post date:** [May 18, 2018, 9:14am UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/6 "2018-05-18T09:14:55Z")

</div>

Admlko

I have not been able to replicate the problem in version 6.2.4.

The results indices are created with a index template perhaps the problem is there. In dev tools can you run the following and share the output please.

```
GET _template/.ml-anomalies-

```

I am presuming you are trying to create the job in the UI can you try creating the job directly via the rest API and report any errors. Again you can do this in dev tools.

```
PUT _xpack/ml/anomaly_detectors/mapping_error_job
{
  "analysis_config": {
    "bucket_span": "15m",
    "influencers": [
      "client_ip.keyword",
      "username.keyword",
      "keywords.keyword"
    ],
    "detectors": [
      {
        "function": "high_count",
        "over_field_name": "username.keyword",
        "partition_field_name": "customer.keyword",
        "exclude_frequent": "all"
      }
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "results_index_name": "brand_new_index"  
}
```

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 18, 2018, 9:39am UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/7 "2018-05-18T09:39:21Z")

</div>

@dkyle  
Thank you for your reply! 🙂

Input:

```auto
GET _template/.ml-anomalies-

```

**[Output here](https://privatebin.net/?0ed74d565bda5d3d#VSoaY9ZgRrtSdfeE67aPrPkDqDYT08vWvUE/cjfmLXg=)**

Input:

```auto
PUT _xpack/ml/anomaly_detectors/mapping_error_job
{
  "analysis_config": {
    "bucket_span": "15m",
    "influencers": [
      "client_ip.keyword",
      "username.keyword",
      "keywords.keyword"
    ],
    "detectors": [
      {
        "function": "high_count",
        "over_field_name": "username.keyword",
        "partition_field_name": "customer.keyword",
        "exclude_frequent": "all"
      }
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "results_index_name": "ml-anomaly-test-mapping-error"  
}

```

Output:

```auto
  "job_id": "mapping_error_job",
  "job_type": "anomaly_detector",
  "job_version": "6.2.4",
  "create_time": 1526635339352,
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "detector_description": """high_count over "username.keyword" partitionfield="customer.keyword" excludefrequent=all""",
        "function": "high_count",
        "over_field_name": "username.keyword",
        "partition_field_name": "customer.keyword",
        "exclude_frequent": "all",
        "rules": [],
        "detector_index": 0
      }
    ],
    "influencers": [
      "client_ip.keyword",
      "username.keyword",
      "keywords.keyword"
    ]
  },
  "analysis_limits": {
    "model_memory_limit": "1024mb"
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_ms"
  },
  "model_snapshot_retention_days": 1,
  "results_index_name": "custom-ml-anomaly-test-mapping-error"
}

```

I can see the job in Machine Learning app in Kibana and it is marked as stopped, but if I click edit job, it says that **Datafeed settings cannot be edited while the datafeed is running.**

If I click "Stop datafeed", I get the red error:

```auto
Could not stop datafeed for mapping_error_job [resource_not_found_exception] No datafeed with id [undefined] exists

```

Makes sense because the API call didn't include datafeed config, but the error could be a bit more informative/less crashy 🙂  
So I guess this is how it is supposed to work, when the job is created through API and not Kibana?

After this, I tried to create same job from Kibana UI and now it magically started to work, I get no errors and the datafeed started to run normally.

It still bothers me though, what broke it and why didn't it work...

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 18, 2018, 9:49am UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/8 "2018-05-18T09:49:01Z")

</div>

Nope, there is something wrong with my datafeed.

Here's the config I'm trying to use:

```auto
  "datafeed_config": {
    "query": {
      "bool": {
        "should": [
          {
            "match_phrase": {
              "event_id": "4608"
            }
          },
          {
            "match_phrase": {
              "event_id": "4609"
            }
          },
          {
            "match_phrase": {
              "event_id": "4634"
            }
          },
          {
            "match_phrase": {
              "event_id": "4647"
            }
          },
          {
            "match_phrase": {
              "event_id": "4778"
            }
          },
          {
            "match_phrase": {
              "event_id": "4779"
            }
          },
          {
            "match_phrase": {
              "event_id": "4800"
            }
          },
          {
            "match_phrase": {
              "event_id": "4801"
            }
          },
          {
            "match_phrase": {
              "event_id": "4802"
            }
          },
          {
            "match_phrase": {
              "event_id": "4803"
            }
          },
          {
            "match_phrase": {
              "event_id": "4624"
            }
          },
          {
            "match_phrase": {
              "event_id": "4625"
            }
          },
          {
            "match_phrase": {
              "event_id": "4648"
            }
          },
          {
            "match_phrase": {
              "event_id": "4672"
            }
          },
          {
            "match_phrase": {
              "event_id": "4768"
            }
          },
          {
            "match_phrase": {
              "event_id": "4769"
            }
          },
          {
            "match_phrase": {
              "event_id": "4770"
            }
          },
          {
            "match_phrase": {
              "event_id": "4771"
            }
          },
          {
            "match_phrase": {
              "event_id": "4772"
            }
          },
          {
            "match_phrase": {
              "event_id": "4773"
            }
          },
          {
            "match_phrase": {
              "event_id": "4774"
            }
          },
          {
            "match_phrase": {
              "event_id": "4775"
            }
          },
          {
            "match_phrase": {
              "event_id": "4776"
            }
          },
          {
            "match_phrase": {
              "event_id": "4777"
            }
          },
          {
            "match_phrase": {
              "event_id": "4778"
            }
          },
          {
            "match_phrase": {
              "event_id": "4779"
            }
          },
          {
            "match_phrase": {
              "event_id": "4820"
            }
          },
          {
            "match_phrase": {
              "event_id": "4821"
            }
          },
          {
            "match_phrase": {
              "event_id": "4822"
            }
          },
          {
            "match_phrase": {
              "event_id": "4823"
            }
          },
          {
            "match_phrase": {
              "event_id": "4824"
            }
          },
          {
            "match_phrase": {
              "event_id": "4825"
            }
          }
        ],
        "minimum_should_match": 1
      }
    },
    "indices": [
      "*-adaudit-*"
    ],
    "types": []
  }

```

What's wrong with this?  
If I add this to the job, I get the original error.  
If I take it away, the job gets created just fine.

**EDIT:**  
Okay, think I got it now.  
So if the job name contains "adaudit", it will be in the result index name too (if Kibana UI is used). So the index pattern _-adaudit-_ will also contain the anomaly result index, and I have a hunch that is the cause for this.  
Let me debug a big more...

---

<div class="post-metadata">

**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\
**Post date:** [May 18, 2018, 10:00am UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/9 "2018-05-18T10:00:57Z")

</div>

Ok that is interesting.

First up I'd suggest the `match_phrase` query isn't the optimal solution here. Have you considered a [Terms Query](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/query-dsl-terms-query.html) for one thing it would be much less verbose as you wouldn't need the bool query

```
 "query": {
        "terms" : { "event_id" : ["4608", "4609", ...]}
    }
```

---

<div class="post-metadata">

**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\
**Post date:** [May 18, 2018, 10:02am UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/10 "2018-05-18T10:02:14Z")

</div>

> [@admlko](#):
>
> EDIT:
> 
> Okay, think I got it now.
> 
> So if the job name contains "adaudit", it will be in the result index name too (if Kibana UI is used). So the index pattern -adaudit- will also contain the anomaly result index, and I have a hunch that is the cause for this.
> 
> Let me debug a big more...

Now it's beginning to make sense 😀

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 18, 2018, 10:11am UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/11 "2018-05-18T10:11:53Z")

</div>

> First up I'd suggest the match\_phrase query isn't the optimal solution here. Have you considered a Terms Query for one thing it would be much less verbose as you wouldn't need the bool query
> 
> ```auto
> "query": {
> "terms" : { "event_id" : ["4608", "4609", ...]}
> }
> 
> ```

Sure, yeah.  
Cheers, I agree that was butt-ugly. Don't know where did it originally came from 😊

> > EDIT:
> > 
> > Okay, think I got it now.
> > 
> > So if the job name contains "adaudit", it will be in the result index name too (if Kibana UI is used). So the index pattern -adaudit- will also contain the anomaly result index, and I have a hunch that is the cause for this.
> 
> Let me debug a big more...  
> Now it's beginning to make sense 😀

Geez!  
I can confirm this was the issue and now it seems to work as expected.

Wow, too much time spent on this issue which once again was quite trivial.

My original idea was to use aliases, because using \*-\* patterns is f-ing ugly and as witnessed, very dangerous or at least error prone.  
But as for now it is the only way to achieve this because [Kibana has a bug which prevents aliases for being used as index pattern](https://github.com/elastic/kibana/issues/16705).  
Have to wait for 6.3...

Anyway, thanks @dkyle a lot, you have been a lifesaver!

---

<div class="post-metadata">

**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\
**Post date:** [May 18, 2018, 10:14am UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/12 "2018-05-18T10:14:46Z")

</div>

My pleasure I'm glad you got to the bottom of it

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 30, 2018, 10:21pm UTC](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100/13 "2018-10-30T22:21:35Z")

</div>