Machine Learning broke somehow

Elastic Stack Elasticsearch
1

Sorry about the topic, but I honestly have no clue what is the issue.

I was using Elastic stack 6.2.2, had four ML jobs and they were running fine.

Then I was adding a new job, and got this error immediately when I clicked save job:

Save failed: [mapper_parsing_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current_type [text], merged_type [ObjectMapper]

Updated the whole stack from 6.2.2 to 6.2.4, same result, but now the old jobs didn't start to work.
They all started and it said that the datafeed was running (also elasticsearch node logs said so), but there was an exclamation mark next to all the jobs, some said that no nodes found and some said something like "no model snapshot found".

I deleted all jobs and all .ml-* indices, but same result:

Save failed: [mapper_parsing_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current_type [text], merged_type [ObjectMapper]

Checked Elasticsearch log from all nodes, no logs regarding this issue.

I would appreciate if someone could help me to get the ML on its feet again :slight_smile:

2

Hi admlko

When a job is created the mappings for the fields used by the job are added to the results index. In this case there must be a clash for the field keywords with an existing mapping. Note the index mappings are held in the cluster state so deleting the indices won't remove the mapping. Take a look a the Mapping API

The simplest fix is to use a dedicated index for the job's results then you won't get the mapping clash. When you create a job expand the Advanced section and check the Use dedicated index box as seen below.

dedicated_index

Was this the only error you saw in the UI? I would have expected a more informative message.

3

Hi dkyle,

I am really glad you replied so fast!

When a job is created the mappings for the fields used by the job are added to the results index. In this case there must be a clash for the field keywords with an existing mapping. Note the index mappings are held in the cluster state so deleting the indices won't remove the mapping. Take a look a the Mapping API

I have defined my own mappings for all indices I store my data. This particular mapping has a field named "keyword" which is stored both in text and keyword. This ML job uses "keywords.keyword".
I have not changed the mapping in a long time, that's why I am a bit lost what has gone wrong.

The simplest fix is to use a dedicated index for the job's results then you won't get the mapping clash. When you create a job expand the Advanced section and check the Use dedicated index box as seen below.

I always use dedicated indices for all ML jobs, exactly because of this. I have had issues before when some indices have the same field but different type.

Save failed: [mapper_parsing_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current_type [text], merged_type [ObjectMapper]
Was this the only error you saw in the UI? I would have expected a more informative message.

Yes, this was the only error in UI. I was really surprised that there were no additional information in Elasticsearch node logs.

Thank you for looking into this :slight_smile:

4

So you are using a dedicated index?

Can you post the index mapping for the results index you are using and the data index. Use GET mapping

Because you can't save the job you probably won't have the JSON configuration but could you describe the job's analysis config and detector functions please. If you are creating an advanced job you can copy this from the edit json tab before saving the job.

5

So you are using a dedicated index?

Yup :slight_smile:

Can you post the index mapping for the results index you are using and the data index. Use GET mapping

Cannot post the results index mapping because it doesn't create one.
I just tried to create new job, here's the JSON config

And when I click "Save":

Save failed: [mapper_parsing_exception] Failed to parse mapping [doc]: mapper [keywords] of different type, current_type [text], merged_type [ObjectMapper]

I am expecting to see .ml-anomalies-custom-adaudit-ml-test index, but it is not created.

Mapping for the data index

6

Admlko

I have not been able to replicate the problem in version 6.2.4.

The results indices are created with a index template perhaps the problem is there. In dev tools can you run the following and share the output please.

GET _template/.ml-anomalies-

I am presuming you are trying to create the job in the UI can you try creating the job directly via the rest API and report any errors. Again you can do this in dev tools.

PUT _xpack/ml/anomaly_detectors/mapping_error_job
{
  "analysis_config": {
    "bucket_span": "15m",
    "influencers": [
      "client_ip.keyword",
      "username.keyword",
      "keywords.keyword"
    ],
    "detectors": [
      {
        "function": "high_count",
        "over_field_name": "username.keyword",
        "partition_field_name": "customer.keyword",
        "exclude_frequent": "all"
      }
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "results_index_name": "brand_new_index"  
}
7

@dkyle
Thank you for your reply! :slight_smile:

Input:

GET _template/.ml-anomalies-

Output here

Input:

PUT _xpack/ml/anomaly_detectors/mapping_error_job
{
  "analysis_config": {
    "bucket_span": "15m",
    "influencers": [
      "client_ip.keyword",
      "username.keyword",
      "keywords.keyword"
    ],
    "detectors": [
      {
        "function": "high_count",
        "over_field_name": "username.keyword",
        "partition_field_name": "customer.keyword",
        "exclude_frequent": "all"
      }
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "results_index_name": "ml-anomaly-test-mapping-error"  
}

Output:

  "job_id": "mapping_error_job",
  "job_type": "anomaly_detector",
  "job_version": "6.2.4",
  "create_time": 1526635339352,
  "analysis_config": {
    "bucket_span": "15m",
    "detectors": [
      {
        "detector_description": """high_count over "username.keyword" partitionfield="customer.keyword" excludefrequent=all""",
        "function": "high_count",
        "over_field_name": "username.keyword",
        "partition_field_name": "customer.keyword",
        "exclude_frequent": "all",
        "rules": [],
        "detector_index": 0
      }
    ],
    "influencers": [
      "client_ip.keyword",
      "username.keyword",
      "keywords.keyword"
    ]
  },
  "analysis_limits": {
    "model_memory_limit": "1024mb"
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_ms"
  },
  "model_snapshot_retention_days": 1,
  "results_index_name": "custom-ml-anomaly-test-mapping-error"
}

I can see the job in Machine Learning app in Kibana and it is marked as stopped, but if I click edit job, it says that Datafeed settings cannot be edited while the datafeed is running.

If I click "Stop datafeed", I get the red error:

Could not stop datafeed for mapping_error_job [resource_not_found_exception] No datafeed with id [undefined] exists

Makes sense because the API call didn't include datafeed config, but the error could be a bit more informative/less crashy :slight_smile:
So I guess this is how it is supposed to work, when the job is created through API and not Kibana?

After this, I tried to create same job from Kibana UI and now it magically started to work, I get no errors and the datafeed started to run normally.

It still bothers me though, what broke it and why didn't it work...

8

Nope, there is something wrong with my datafeed.

Here's the config I'm trying to use:

  "datafeed_config": {
    "query": {
      "bool": {
        "should": [
          {
            "match_phrase": {
              "event_id": "4608"
            }
          },
          {
            "match_phrase": {
              "event_id": "4609"
            }
          },
          {
            "match_phrase": {
              "event_id": "4634"
            }
          },
          {
            "match_phrase": {
              "event_id": "4647"
            }
          },
          {
            "match_phrase": {
              "event_id": "4778"
            }
          },
          {
            "match_phrase": {
              "event_id": "4779"
            }
          },
          {
            "match_phrase": {
              "event_id": "4800"
            }
          },
          {
            "match_phrase": {
              "event_id": "4801"
            }
          },
          {
            "match_phrase": {
              "event_id": "4802"
            }
          },
          {
            "match_phrase": {
              "event_id": "4803"
            }
          },
          {
            "match_phrase": {
              "event_id": "4624"
            }
          },
          {
            "match_phrase": {
              "event_id": "4625"
            }
          },
          {
            "match_phrase": {
              "event_id": "4648"
            }
          },
          {
            "match_phrase": {
              "event_id": "4672"
            }
          },
          {
            "match_phrase": {
              "event_id": "4768"
            }
          },
          {
            "match_phrase": {
              "event_id": "4769"
            }
          },
          {
            "match_phrase": {
              "event_id": "4770"
            }
          },
          {
            "match_phrase": {
              "event_id": "4771"
            }
          },
          {
            "match_phrase": {
              "event_id": "4772"
            }
          },
          {
            "match_phrase": {
              "event_id": "4773"
            }
          },
          {
            "match_phrase": {
              "event_id": "4774"
            }
          },
          {
            "match_phrase": {
              "event_id": "4775"
            }
          },
          {
            "match_phrase": {
              "event_id": "4776"
            }
          },
          {
            "match_phrase": {
              "event_id": "4777"
            }
          },
          {
            "match_phrase": {
              "event_id": "4778"
            }
          },
          {
            "match_phrase": {
              "event_id": "4779"
            }
          },
          {
            "match_phrase": {
              "event_id": "4820"
            }
          },
          {
            "match_phrase": {
              "event_id": "4821"
            }
          },
          {
            "match_phrase": {
              "event_id": "4822"
            }
          },
          {
            "match_phrase": {
              "event_id": "4823"
            }
          },
          {
            "match_phrase": {
              "event_id": "4824"
            }
          },
          {
            "match_phrase": {
              "event_id": "4825"
            }
          }
        ],
        "minimum_should_match": 1
      }
    },
    "indices": [
      "*-adaudit-*"
    ],
    "types": []
  }

What's wrong with this?
If I add this to the job, I get the original error.
If I take it away, the job gets created just fine.

EDIT:
Okay, think I got it now.
So if the job name contains "adaudit", it will be in the result index name too (if Kibana UI is used). So the index pattern -adaudit- will also contain the anomaly result index, and I have a hunch that is the cause for this.
Let me debug a big more...

9

Ok that is interesting.

First up I'd suggest the match_phrase query isn't the optimal solution here. Have you considered a Terms Query for one thing it would be much less verbose as you wouldn't need the bool query

 "query": {
        "terms" : { "event_id" : ["4608", "4609", ...]}
    }
10

Now it's beginning to make sense :grinning:

11

First up I'd suggest the match_phrase query isn't the optimal solution here. Have you considered a Terms Query for one thing it would be much less verbose as you wouldn't need the bool query

 "query": {
        "terms" : { "event_id" : ["4608", "4609", ...]}
    }

Sure, yeah.
Cheers, I agree that was butt-ugly. Don't know where did it originally came from :blush:

EDIT:

Okay, think I got it now.

So if the job name contains "adaudit", it will be in the result index name too (if Kibana UI is used). So the index pattern -adaudit- will also contain the anomaly result index, and I have a hunch that is the cause for this.

Let me debug a big more...
Now it's beginning to make sense :grinning:

Geez!
I can confirm this was the issue and now it seems to work as expected.

Wow, too much time spent on this issue which once again was quite trivial.

My original idea was to use aliases, because using *-* patterns is f-ing ugly and as witnessed, very dangerous or at least error prone.
But as for now it is the only way to achieve this because Kibana has a bug which prevents aliases for being used as index pattern.
Have to wait for 6.3...

Anyway, thanks @dkyle a lot, you have been a lifesaver!

12

My pleasure I'm glad you got to the bottom of it

2 Likes