# Mapping issue when creating ML job **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [September 9, 2019, 3:31pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728 "2019-09-09T15:31:30Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 9, 2019, 3:31pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/1 "2019-09-09T15:31:30Z")
I'm running Elastic Stack 6.8.1 and trying to recreate a ML job. It has been working fine for a couple of years now, but I deleted the old job and tried to create it again, it doesn't allow it to be created. The error is: > Save failed: [mapper\_parsing\_exception] Failed to parse mapping [doc]: mapper [Packet-Type] of different type, current\_type [text], merged\_type [ObjectMapper] I understand that it means that there is at least one index which has different mapping for Packet-Type (actually, Packet-Type.raw) field than all the other indices. I have gone through the indices and templates but everywhere the Packet-Type has a raw field and it is always keyword type. The Packet-Type field is defined as follows: ``` "Packet-Type" : { "type" : "text", "fields" : { "raw" : { "type" : "keyword" } } } ``` Here's my job config: ``` { "job_id": "logins", "groups": [], "results_retention_days": 180, "analysis_config": { "bucket_span": "15m", "detectors": [ { "detector_description": "high_count over \"Fully-Qualified-Distinguished-Name.raw\" partitionfield=customer excludefrequent=all", "function": "high_count", "over_field_name": "Fully-Qualified-Distinguished-Name.raw", "partition_field_name": "customer", "exclude_frequent": "all", "detector_index": 0 } ], "influencers": [ "Fully-Qualified-Distinguished-Name.raw", "Reason-Code", "Packet-Type.raw" ] }, "data_description": { "time_field": "@timestamp" }, "results_index_name": "custom-logins", "datafeed_config": { "query": { "match_all": {} }, "indices": [ "all-logins" ], "types": [] } } ``` What am I missing? ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 9, 2019, 7:36pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/2 "2019-09-09T19:36:02Z")
Seems like you've had this problem before, but resolved it back then? > [@Machine Learning broke somehow](https://discuss.elastic.co/t/machine-learning-broke-somehow/132100): > > Sorry about the topic, but I honestly have no clue what is the issue. I was using Elastic stack 6.2.2, had four ML jobs and they were running fine. Then I was adding a new job, and got this error immediately when I clicked save job: Save failed: [mapper\_parsing\_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current\_type [text], merged\_type [ObjectMapper] Updated the whole stack from 6.2.2 to 6.2.4, same result, but now the old jobs didn't start to work. The… ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 10, 2019, 5:06am UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/3 "2019-09-10T05:06:58Z")
Yeah, I am in the middle of migrating from `*-*` index patterns to using aliases. I have checked the indices returned by the alias that their mappings are all same. But it still sounds like my bad, so can you give me the query which I should run in order to list all index mappings which has that alias? Just to be sure 🙂 EDIT: Would be nice see the mismatching index names directly in the error message. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 10, 2019, 1:14pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/4 "2019-09-10T13:14:42Z")
If you have been running ML for years then you must also have a support contract with us. Have you considered opening a support ticket? ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 10, 2019, 2:30pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/5 "2019-09-10T14:30:13Z")
Lol no support. Just platinum from the time when your subscription offering was really messy. Just renewed it for yet another two years. I don't expect any SLA... ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 10, 2019, 3:24pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/6 "2019-09-10T15:24:35Z")
So if anyone is willing to give a hand, here's how I have tried to check the mappings: > GET all-logins/\_mapping/field/Packet-Type.raw ``` { "customer1-logins-2019.09" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer1-logins-2019.08" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer3-logins-2019" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer4-logins-2019" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer5-logins-2019" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer1-logins-2019.01" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer1-logins-2019.03" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer1-logins-2019.02" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer1-logins-2019.05" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer2-logins-2019" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer1-logins-2019.04" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer1-logins-2019.07" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer6-logins-2019" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer1-logins-2019.06" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer7-logins-2019" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } }, "customer8-logins-2019" : { "mappings" : { "doc" : { "Packet-Type.raw" : { "full_name" : "Packet-Type.raw", "mapping" : { "raw" : { "type" : "keyword" } } } } } } } ``` Based on the error, I was expecting to see type `text` but not cigar. I also don't understand why is it complaining about `Packet-Type` field, when only `Packet-Type.raw` field is defined in the influencers. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 10, 2019, 9:07pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/7 "2019-09-10T21:07:51Z")
You should check the mappings in the `.ml-anomalies-*` indices (seemingly specifically the `.ml-anomalies-custom-logins` index in your case because your job sets an explicit `results_index_name`) That's where the conflict lies. If you don't need old results from the "old" job, you can just delete `.ml-anomalies-custom-logins` and start over. Or, set `results_index_name` to something new, like `"custom-logins-new"` ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 11, 2019, 4:28am UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/8 "2019-09-11T04:28:02Z")
There is no `.ml-anomalies-custom-logins` index anymore, it got deleted when I removed the old job. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 11, 2019, 10:50am UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/9 "2019-09-11T10:50:41Z")
Ok - my theory that you already had the index was wrong - therefore the conflict is in the `.ml-anomalies-shared` index if you are not explicitly setting a `results_index_name` on job creation. ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 11, 2019, 11:02am UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/10 "2019-09-11T11:02:30Z")
There is no `.ml-anomalies-shared` index, all the jobs - including the one I am trying to create now - has the `results_index_name` set. Here are all the .ml indices in the system: ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/4/c41fd8f27829ab08ec27bd112b71be87095793ff.png) ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 11, 2019, 12:13pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/11 "2019-09-11T12:13:48Z")
Thanks for the names of the existing .ml indices - this helps. Now can you please show the exact command/API call you're using to try to create the new job? ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 11, 2019, 12:24pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/12 "2019-09-11T12:24:49Z")
I am using Kibana's wizard. I go to ML app, select create new job, advanced job and paste the aforementioned JSON. Validation goes through, but creation results the error. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 11, 2019, 2:35pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/13 "2019-09-11T14:35:28Z")
ok - let's simplify this a little and only use the API. Go to the Dev Tools console and try this: ```auto PUT _xpack/ml/anomaly_detectors/logins { "analysis_config": { "bucket_span": "15m", "detectors": [ { "detector_description": "high_count over \"Fully-Qualified-Distinguished-Name.raw\" partitionfield=customer excludefrequent=all", "function": "high_count", "over_field_name": "Fully-Qualified-Distinguished-Name.raw", "partition_field_name": "customer", "exclude_frequent": "all", "detector_index": 0 } ], "influencers": [ "Fully-Qualified-Distinguished-Name.raw", "Reason-Code", "Packet-Type.raw" ] }, "data_description": { "time_field": "@timestamp" }, "results_index_name": "custom-logins" } ``` If this works, this should properly create an index called `.ml-anomalies-custom-logins` to which you could verify it creates the proper mappings ```auto GET .ml-anomalies-custom-logins/_mapping ``` Can you try? And, if that works, try creating the companion datafeed: ```auto PUT _xpack/ml/datafeeds/datafeed-logins/ { "job_id": "logins", "indices": [ "all-logins" ], "query": { "match_all": { } } } ``` ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 11, 2019, 3:21pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/14 "2019-09-11T15:21:48Z")
> ok - let's simplify this a little and only use the API. Go to the Dev Tools console and try this: > > ```auto > PUT _xpack/ml/anomaly_detectors/logins > { > "analysis_config": { > "bucket_span": "15m", > "detectors": [ > { > "detector_description": "high_count over \"Fully-Qualified-Distinguished-Name.raw\" partitionfield=customer excludefrequent=all", > "function": "high_count", > "over_field_name": "Fully-Qualified-Distinguished-Name.raw", > "partition_field_name": "customer", > "exclude_frequent": "all", > "detector_index": 0 > } > ], > "influencers": [ > "Fully-Qualified-Distinguished-Name.raw", > "Reason-Code", > "Packet-Type.raw" > ] > }, > "data_description": { > "time_field": "@timestamp" > }, > "results_index_name": "custom-logins" > } > > ``` Results: ``` { "error": { "root_cause": [ { "type": "mapper_parsing_exception", "reason": "Failed to parse mapping [doc]: mapper [Packet-Type] of different type, current_type [text], merged_type [ObjectMapper]" } ], "type": "mapper_parsing_exception", "reason": "Failed to parse mapping [doc]: mapper [Packet-Type] of different type, current_type [text], merged_type [ObjectMapper]", "caused_by": { "type": "illegal_argument_exception", "reason": "mapper [Packet-Type] of different type, current_type [text], merged_type [ObjectMapper]" } }, "status": 400 } ``` ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 11, 2019, 3:57pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/15 "2019-09-11T15:57:38Z")
Ok - great - now can you execute: ```auto GET _cat/indices/.ml* ``` ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 11, 2019, 3:58pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/16 "2019-09-11T15:58:45Z")
``` green open .ml-config 38aeg9N1RDW2KtvKy6mPPQ 1 1 4 0 96.7kb 48.3kb green open .ml-annotations-6 YqvaO-CZQ66j6pWtMFFSZA 1 1 10 0 119.9kb 59.9kb green open .ml-state IIm0JWSESI-R2xrRyc8zTw 5 1 4809 351 64.4gb 32.1gb green open .ml-anomalies-custom-sonic-remote-logins rpsj5lGHQeitR116tiPrIg 5 1 4456 109 2.3mb 1.1mb green open .ml-anomalies-custom-active-directory-logins mRNlejceQ8SBa8dvL9_SuQ 5 1 5411468 197709 1.5gb 808.4mb green open .ml-notifications JEqh7S4vSNmk8E0ZdlzAcA 1 1 17318 0 8.8mb 4.4mb ``` Still the same... ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 11, 2019, 4:32pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/17 "2019-09-11T16:32:14Z")
Ok great - now please try this: ```auto GET .ml-anomalies-*/_mapping/field/Packet-Type ``` or even ```auto GET .ml-*/_mapping/field/Packet-Type ``` there has to be something leftover from the previous version you were on somewhere. What version did you upgrade from? ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 11, 2019, 4:41pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/18 "2019-09-11T16:41:15Z")
> Ok great - now please try this: > > ```auto > GET .ml-anomalies-*/_mapping/field/Packet-Type > > ``` Result: ``` #! Deprecation: [types removal] The parameter include_type_name should be explicitly specified in get field mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions. { ".ml-anomalies-custom-sonic-remote-logins" : { "mappings" : { } }, ".ml-anomalies-custom-active-directory-logins" : { "mappings" : { } } } ``` > or even > > ```auto > GET .ml-*/_mapping/field/Packet-Type > > ``` Result: ``` #! Deprecation: [types removal] The parameter include_type_name should be explicitly specified in get field mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions. { ".ml-state" : { "mappings" : { } }, ".ml-anomalies-custom-active-directory-logins" : { "mappings" : { } }, ".ml-anomalies-custom-sonic-remote-logins" : { "mappings" : { } }, ".ml-annotations-6" : { "mappings" : { } }, ".ml-notifications" : { "mappings" : { } }, ".ml-config" : { "mappings" : { } } } ``` I think the version was 6.3.x ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 11, 2019, 4:48pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/19 "2019-09-11T16:48:08Z")
Hmmm...ok - let's keep going to see if there is a residual index alias somewhere: ```auto GET _cat/aliases/.ml*?v ``` ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [September 11, 2019, 4:49pm UTC](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728/20 "2019-09-11T16:49:15Z")
> Hmmm...ok - let's keep going to see if there is a residual index alias somewhere: > > ```auto > GET _cat/aliases/.ml*?v > > ``` Result: ``` alias index filter routing.index routing.search .ml-anomalies-.write-active-directory-logins .ml-anomalies-custom-active-directory-logins - - - .ml-anomalies-active-directory-logins .ml-anomalies-custom-active-directory-logins * - - .ml-state-write .ml-state - - - .ml-anomalies-.write-sonic-remote-logins .ml-anomalies-custom-sonic-remote-logins - - - .ml-anomalies-sonic-remote-logins .ml-anomalies-custom-sonic-remote-logins * - - .ml-annotations-read .ml-annotations-6 - - - .ml-annotations-write .ml-annotations-6 - - - ``` [Next page](https://discuss.elastic.co/t/mapping-issue-when-creating-ml-job/198728.md?page=2)