Mapping issue when creating ML job

Elastic Stack Elasticsearch
1

I'm running Elastic Stack 6.8.1 and trying to recreate a ML job.
It has been working fine for a couple of years now, but I deleted the old job and tried to create it again, it doesn't allow it to be created.

The error is:

Save failed: [mapper_parsing_exception] Failed to parse mapping [doc]: mapper [Packet-Type] of different type, current_type [text], merged_type [ObjectMapper]

I understand that it means that there is at least one index which has different mapping for Packet-Type (actually, Packet-Type.raw) field than all the other indices. I have gone through the indices and templates but everywhere the Packet-Type has a raw field and it is always keyword type.

The Packet-Type field is defined as follows:

"Packet-Type" : {
  "type" : "text",
  "fields" : {
    "raw" : {
      "type" : "keyword"
    }
  }
}

Here's my job config:

{
  "job_id": "logins",
  "groups": [],
  "results_retention_days": 180,
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "detector_description": "high_count over \"Fully-Qualified-Distinguished-Name.raw\" partitionfield=customer excludefrequent=all",
        "function": "high_count",
        "over_field_name": "Fully-Qualified-Distinguished-Name.raw",
        "partition_field_name": "customer",
        "exclude_frequent": "all",
        "detector_index": 0
      }
    ],
    "influencers": [
      "Fully-Qualified-Distinguished-Name.raw",
      "Reason-Code",
      "Packet-Type.raw"
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "results_index_name": "custom-logins",
  "datafeed_config": {
    "query": {
      "match_all": {}
    },
    "indices": [
      "all-logins"
    ],
    "types": []
  }
}

What am I missing?

2

Seems like you've had this problem before, but resolved it back then?

3

Yeah, I am in the middle of migrating from *-* index patterns to using aliases.
I have checked the indices returned by the alias that their mappings are all same. But it still sounds like my bad, so can you give me the query which I should run in order to list all index mappings which has that alias? Just to be sure :slight_smile:

EDIT: Would be nice see the mismatching index names directly in the error message.

4

If you have been running ML for years then you must also have a support contract with us. Have you considered opening a support ticket?

1 Like
5

Lol no support. Just platinum from the time when your subscription offering was really messy. Just renewed it for yet another two years. I don't expect any SLA...

6

So if anyone is willing to give a hand, here's how I have tried to check the mappings:

GET all-logins/_mapping/field/Packet-Type.raw

{
  "customer1-logins-2019.09" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer1-logins-2019.08" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer3-logins-2019" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer4-logins-2019" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer5-logins-2019" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer1-logins-2019.01" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer1-logins-2019.03" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer1-logins-2019.02" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer1-logins-2019.05" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer2-logins-2019" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer1-logins-2019.04" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer1-logins-2019.07" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer6-logins-2019" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer1-logins-2019.06" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer7-logins-2019" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  },
  "customer8-logins-2019" : {
    "mappings" : {
      "doc" : {
        "Packet-Type.raw" : {
          "full_name" : "Packet-Type.raw",
          "mapping" : {
            "raw" : {
              "type" : "keyword"
            }
          }
        }
      }
    }
  }
}

Based on the error, I was expecting to see type text but not cigar.

I also don't understand why is it complaining about Packet-Type field, when only Packet-Type.raw field is defined in the influencers.

7

You should check the mappings in the .ml-anomalies-* indices (seemingly specifically the .ml-anomalies-custom-logins index in your case because your job sets an explicit results_index_name)

That's where the conflict lies. If you don't need old results from the "old" job, you can just delete .ml-anomalies-custom-logins and start over. Or, set results_index_name to something new, like "custom-logins-new"

8

There is no .ml-anomalies-custom-logins index anymore, it got deleted when I removed the old job.

9

Ok - my theory that you already had the index was wrong - therefore the conflict is in the .ml-anomalies-shared index if you are not explicitly setting a results_index_name on job creation.

10

There is no .ml-anomalies-shared index, all the jobs - including the one I am trying to create now - has the results_index_name set.
Here are all the .ml indices in the system:

image
image.png2400×1094 189 KB

11

Thanks for the names of the existing .ml indices - this helps. Now can you please show the exact command/API call you're using to try to create the new job?

12

I am using Kibana's wizard. I go to ML app, select create new job, advanced job and paste the aforementioned JSON. Validation goes through, but creation results the error.

13

ok - let's simplify this a little and only use the API. Go to the Dev Tools console and try this:

PUT _xpack/ml/anomaly_detectors/logins
{
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "detector_description": "high_count over \"Fully-Qualified-Distinguished-Name.raw\" partitionfield=customer excludefrequent=all",
        "function": "high_count",
        "over_field_name": "Fully-Qualified-Distinguished-Name.raw",
        "partition_field_name": "customer",
        "exclude_frequent": "all",
        "detector_index": 0
      }
    ],
    "influencers": [
      "Fully-Qualified-Distinguished-Name.raw",
      "Reason-Code",
      "Packet-Type.raw"
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "results_index_name": "custom-logins"
}

If this works, this should properly create an index called .ml-anomalies-custom-logins to which you could verify it creates the proper mappings

GET .ml-anomalies-custom-logins/_mapping

Can you try?

And, if that works, try creating the companion datafeed:

PUT _xpack/ml/datafeeds/datafeed-logins/
{
  "job_id": "logins",
  "indices": [
    "all-logins"
  ],
      "query": {
        "match_all": {
        }
      }
}
14

ok - let's simplify this a little and only use the API. Go to the Dev Tools console and try this:

PUT _xpack/ml/anomaly_detectors/logins
{
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "detector_description": "high_count over \"Fully-Qualified-Distinguished-Name.raw\" partitionfield=customer excludefrequent=all",
        "function": "high_count",
        "over_field_name": "Fully-Qualified-Distinguished-Name.raw",
        "partition_field_name": "customer",
        "exclude_frequent": "all",
        "detector_index": 0
      }
    ],
    "influencers": [
      "Fully-Qualified-Distinguished-Name.raw",
      "Reason-Code",
      "Packet-Type.raw"
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "results_index_name": "custom-logins"
}

Results:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Failed to parse mapping [doc]: mapper [Packet-Type] of different type, current_type [text], merged_type [ObjectMapper]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Failed to parse mapping [doc]: mapper [Packet-Type] of different type, current_type [text], merged_type [ObjectMapper]",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "mapper [Packet-Type] of different type, current_type [text], merged_type [ObjectMapper]"
    }
  },
  "status": 400
}
15

Ok - great - now can you execute:

GET _cat/indices/.ml*
16 
green open .ml-config                                   38aeg9N1RDW2KtvKy6mPPQ 1 1       4      0  96.7kb  48.3kb
green open .ml-annotations-6                            YqvaO-CZQ66j6pWtMFFSZA 1 1      10      0 119.9kb  59.9kb
green open .ml-state                                    IIm0JWSESI-R2xrRyc8zTw 5 1    4809    351  64.4gb  32.1gb
green open .ml-anomalies-custom-sonic-remote-logins     rpsj5lGHQeitR116tiPrIg 5 1    4456    109   2.3mb   1.1mb
green open .ml-anomalies-custom-active-directory-logins mRNlejceQ8SBa8dvL9_SuQ 5 1 5411468 197709   1.5gb 808.4mb
green open .ml-notifications                            JEqh7S4vSNmk8E0ZdlzAcA 1 1   17318      0   8.8mb   4.4mb

Still the same...

17

Ok great - now please try this:

GET .ml-anomalies-*/_mapping/field/Packet-Type

or even

GET .ml-*/_mapping/field/Packet-Type

there has to be something leftover from the previous version you were on somewhere. What version did you upgrade from?

18

Ok great - now please try this:

GET .ml-anomalies-*/_mapping/field/Packet-Type

Result:

#! Deprecation: [types removal] The parameter include_type_name should be explicitly specified in get field mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions.
{
  ".ml-anomalies-custom-sonic-remote-logins" : {
    "mappings" : { }
  },
  ".ml-anomalies-custom-active-directory-logins" : {
    "mappings" : { }
  }
}

or even

GET .ml-*/_mapping/field/Packet-Type

Result:

#! Deprecation: [types removal] The parameter include_type_name should be explicitly specified in get field mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions.
{
  ".ml-state" : {
    "mappings" : { }
  },
  ".ml-anomalies-custom-active-directory-logins" : {
    "mappings" : { }
  },
  ".ml-anomalies-custom-sonic-remote-logins" : {
    "mappings" : { }
  },
  ".ml-annotations-6" : {
    "mappings" : { }
  },
  ".ml-notifications" : {
    "mappings" : { }
  },
  ".ml-config" : {
    "mappings" : { }
  }
}

I think the version was 6.3.x

19

Hmmm...ok - let's keep going to see if there is a residual index alias somewhere:

GET _cat/aliases/.ml*?v
20

Hmmm...ok - let's keep going to see if there is a residual index alias somewhere:

GET _cat/aliases/.ml*?v

Result:

alias                                        index                                        filter routing.index routing.search
.ml-anomalies-.write-active-directory-logins .ml-anomalies-custom-active-directory-logins -      -             -
.ml-anomalies-active-directory-logins        .ml-anomalies-custom-active-directory-logins *      -             -
.ml-state-write                              .ml-state                                    -      -             -
.ml-anomalies-.write-sonic-remote-logins     .ml-anomalies-custom-sonic-remote-logins     -      -             -
.ml-anomalies-sonic-remote-logins            .ml-anomalies-custom-sonic-remote-logins     *      -             -
.ml-annotations-read                         .ml-annotations-6                            -      -             -
.ml-annotations-write                        .ml-annotations-6                            -      -             -