Mapping definition for [message] has unsupported parameters: [ignore_above : 1024]

Elastic Stack Elasticsearch
1

It's a new week, and seemingly randomly, this week's log index is failing to be created.

Logged from filebeat is:

{"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: Mapping definition for [message] has unsupported parameters: [ignore_above : 1024]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [message] has unsupported parameters: [ignore_above : 1024]"}}

I've checked and have 3 (identical-looking) index templates, as has been the case for a few months without issue. The filebeat version is a little out of date (6.3.2), but has been working fine up to this point.

The "message" property seems to potentially be covered in the template in two ways:

Dynamic template mapping:

        "strings_as_keyword": {
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }

General field mapping:

              "message": {
                "type": "text",
                "norms": false
              },

I'm just using the "raw JSON" mode for sending the contents of text files with one JSON document per row to ES, so it's not an app-specific type of document.

All I can think is that the general field mapping is only overriding the "type" and "norms" prop or something? The dynamic template may be a red herring, I don't know...

But how can I diagnose and fix this issue? A few 100ks of events are queuing up :slight_smile:

Any help appreciated..

Edit: the response to GET /_template: https://gist.github.com/kierenj/a91df51630b1d06798a105e7a66eb5dd . Index name is filebeat-rr-logs-6.3.2-2018.44 (or would be, if it would create)

2

Hi @Kieren_Johnstone,

I am not sure about what can be the issue, but by now I see that the index templates you have installed don't include the version:

  "filebeat-rr-logs": {
    "order": 1,
    "index_patterns": [
      "filebeat-rr-logs-*"
    ],
  ...
  "filebeat-rr": {
    "order": 1,
    "index_patterns": [
      "filebeat-rr-*"
    ],

This can make different versions to store events in the same index, what can lead to unexpected mapping issues.

Is this intended? How did you install your index templates?

3

I installed the filebeat templates manually, probably a few times with slightly different settings. I read that the version field was ignored and only for external template management systems? arent multiple index templates ehich match the pattern just combined? They all look the same to me so wouldnt that have no effect?

4

Would appreciate any help at all, I'm fairly desperate and our production logging server is backing up very quickly indeed.

5

Can anyone help at all? Or tell me where I can find help?

(The elastic paid support plan's basic cluster costing requirement is $15.5k commitment, or consultancy starts at £3,600 with a minimum of 4 days. I'm at your mercy!)

6

I think this can be more an issue with Elasticsearch than with Beats, I am going to move this topic to the ES category.

Btw, what is the ES version you are using?

7

Thanks :slight_smile: My ES version is 6.4.2.

I carefully deleted two of the duplicate index templates and the issue seems to be resolved.

(No idea how it worked for the previous 43 weeks..)

8

I'm glad to read that you found a workaround.
In any case consider to add the filebeat version to the index patterns, specially if you use different versions at the same time.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.