# Ignore\_malformed fires mapper\_parsing\_exception for keyword type on 7.10.1 (it worked on 7.1.0)

**URL:** <https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142>\
**Category:** Elasticsearch\
**Created:** [December 18, 2020, 11:09pm UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142 "2020-12-18T23:09:47Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![DevFullStack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devfullstack/32/77057_2.png) [@DevFullStack](https://discuss.elastic.co/u/DevFullStack)\
**Post date:** [December 18, 2020, 11:09pm UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/1 "2020-12-18T23:09:47Z")

</div>

This may be a breaking change in **v.7.4+** that apparently was [not documented](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/breaking-changes-7.4.html). Any thoughts on how to solve this issue with an upgrade on a logs analysis use case?

The docs about the `ignore_malformed` mapping parameter have some differences when comparing versions [7.3](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/ignore-malformed.html#ignore-malformed) and [7.4](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/ignore-malformed.html#ignore-malformed). From **7.4** on it says:

> The `ignore_malformed` setting is currently supported by the following mapping types: Numeric, Date, Date nanoseconds, Geo-point, Geo-shape, IP

– which excludes other types such as `keyword`.

Trying to index a same document works on `7.1.0` and fails on `7.10.1` (and also `7.8.1`, both upgraded from `7.1.0`), even though the mappings are nearly the same. The `mappings.dynamic_templates` config is:

```javascript
{
  "template_stdField" : {
    "path_match" : "*",
    "mapping" : {
      "ignore_malformed" : true,
      "type" : "keyword"
    }
  }
}

```

Running

```
curl -X PUT "localhost:9200/my-index/notes/2" -H 'Content-Type: application/json' -d'
{
  "text": "hello world",
  "mykeyword": "hello world again"
}
'

```

on `v7.1.0` returns:

```javascript
{
  "_index": "my-index",
  "_type": "notes",
  "_id": "2",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 6,
  "_primary_term": 6
}

```

however, on on `v7.10.1`, the same command returns:

```javascript
{
  "error": {
    "root_cause": [{
      "type": "mapper_parsing_exception",
      "reason": "unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]"
    }],
    "type": "mapper_parsing_exception",
    "reason": "unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]"
  },
  "status": 400
}

```

> **Error trace in 7.10.1**
>
> Here is the output when adding `?error_trace` to the previous command:
> 
> ```javascript
> {
> "error": {
> "root_cause": [{
> "type": "mapper_parsing_exception",
> "reason": "unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]",
> "stack_trace": "MapperParsingException[unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]]\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$Builder.parse(ParametrizedFieldMapper.java:614)\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:662)\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:647)\n\tat org.elasticsearch.index.mapper.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:278)\n\tat org.elasticsearch.index.mapper.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:251)\n\tat org.elasticsearch.index.mapper.DocumentParser.createBuilderFromDynamicValue(DocumentParser.java:704)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:765)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:620)\n\tat org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:424)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:227)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:803)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:780)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:752)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:285)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:175)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:220)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:126)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:85)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:179)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"
> }],
> "type": "mapper_parsing_exception",
> "reason": "unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]",
> "stack_trace": "MapperParsingException[unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]]\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$Builder.parse(ParametrizedFieldMapper.java:614)\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:662)\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:647)\n\tat org.elasticsearch.index.mapper.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:278)\n\tat org.elasticsearch.index.mapper.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:251)\n\tat org.elasticsearch.index.mapper.DocumentParser.createBuilderFromDynamicValue(DocumentParser.java:704)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:765)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:620)\n\tat org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:424)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:227)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:803)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:780)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:752)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:285)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:175)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:220)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:126)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:85)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:179)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"
> },
> "status": 400
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![DevFullStack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devfullstack/32/77057_2.png) [@DevFullStack](https://discuss.elastic.co/u/DevFullStack)\
**Post date:** [December 18, 2020, 11:10pm UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/2 "2020-12-18T23:10:16Z")

</div>

**Index mapping & settings details**  
Running

```
curl "localhost:9200/my-index?pretty"

```

on `v7.1.0` returns (simplified)

```javascript
{
  "my-index" : {
    "mappings" : {
      "dynamic_templates" : [
        {
          "template_text" : {
            "path_match" : "*text",
            "mapping" : {
              "fielddata" : true,
              "type" : "text"
            }
          }
        },
        /* ... */
        {
          "template_object" : {
            "path_match" : "*",
            "match_mapping_type" : "object",
            "mapping" : {
              "doc_values" : true,
              "type" : "object"
            }
          }
        },
        {
          "template_stdField" : {
            "path_match" : "*",
            "mapping" : {
              "ignore_malformed" : true,
              "type" : "keyword"
            }
          }
        }
      ],
      "properties" : {
        "app" : {
          "type" : "keyword"
        },
        /* ... */
        "text" : {
          "type" : "text",
          "fielddata" : true
        }

      }
    },
    "settings" : {
      "index" : {
        "mapping" : {
          /* ... */
          "ignore_malformed" : "true"
        },
        /* ... */
        "version" : {
          "created" : "7010099"
        }
      }
    }
  }
}

```

A similar result is seen on `v7.10.1`, except from the `version` field:

```javascript
"version" : {
  "created" : "7010099",
  "upgraded" : "7100199"
}

```

---

<div class="post-metadata">

**Author:** ![DevFullStack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devfullstack/32/77057_2.png) [@DevFullStack](https://discuss.elastic.co/u/DevFullStack)\
**Post date:** [December 18, 2020, 11:29pm UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/3 "2020-12-18T23:29:51Z")

</div>

Maybe this is related to [7.10 no longer accepts boost mapping on some properties #64982](https://github.com/elastic/elasticsearch/issues/64982) and [Unused boost parameter should not throw mapping exception #64999](https://github.com/elastic/elasticsearch/pull/64999)

---

<div class="post-metadata">

**Author:** ![DevFullStack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devfullstack/32/77057_2.png) [@DevFullStack](https://discuss.elastic.co/u/DevFullStack)\
**Post date:** [December 19, 2020, 2:25am UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/4 "2020-12-19T02:25:48Z")

</div>

Using the [Put mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-put-mapping.html) to update the invalid template should fix this issue. Note that we need to repeat all the previous [dynamic templates](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/dynamic-templates.html) in the proper order so they don't get removed.

```auto
curl -X PUT "localhost:9200/my-index/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
    "dynamic_templates" : [
      /* ... repeat previous templates ... */
      {
        "template_stdField" : {
          "path_match" : "*",
          "mapping" : {
            "type" : "keyword"
          }
        }
      }
    ]

}
'

```

IMHO on the upgrade scenario, instead of throwing an exception without indexing the document, a warning message should be logged and the indexing operation continued.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 19, 2020, 8:35am UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/5 "2020-12-19T08:35:18Z")

</div>

The steps above are incomplete which makes then hard to follow, but this sounds like a bug (at least it's a bug not to document this change).

Could you write out the full instructions that someone would need to run on an empty cluster to reproduce this? If so, please report it on GitHub.

---

<div class="post-metadata">

**Author:** ![DevFullStack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devfullstack/32/77057_2.png) [@DevFullStack](https://discuss.elastic.co/u/DevFullStack)\
**Post date:** [December 21, 2020, 8:08pm UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/6 "2020-12-21T20:08:13Z")

</div>

> [@DavidTurner](#):
>
> The steps above are incomplete which makes then hard to follow

Thanks for your comment David! I apologize for perhaps omitting essential details. On the reply:

> [@DevFullStack](#):
>
> Index mapping & settings details

I tried to put the minimum subset of my mappings that I believe are needed to reproduce the error. Which option are you looking for?

1. the complete set of my mappings,
2. some other index info that I didn't post (pls let me know which one is missing)
3. a series of REST calls that  
**(a)** create an empty index,  
**(b)** set the `dynamic_templates` mapping with one `ignore_malformed` keyword match,  
**(c)** try to index a document to get the error

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 21, 2020, 10:48pm UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/7 "2020-12-21T22:48:23Z")

</div>

I tried following your instructions as best I could and I didn't see what you are describing. Here's a transcript of the exact API calls, and responses, run on a brand-new empty 7.1.0 cluster:

```auto
GET /?filter_path=version.number

# 200 OK
# {
# "version": {
# "number": "7.1.0"
# }
# }

PUT /my_index
{
  "mappings": {
    "dynamic_templates": [
      {
        "template_stdField": {
          "mapping": {
            "ignore_malformed": true,
            "type": "keyword"
          },
          "path_match": "*"
        }
      }
    ]
  }
}

# 200 OK
# {
# "shards_acknowledged": true,
# "acknowledged": true,
# "index": "my_index"
# }

GET /my_index

# 200 OK
# {
# "my_index": {
# "mappings": {
# "dynamic_templates": [
# {
# "template_stdField": {
# "mapping": {
# "ignore_malformed": true,
# "type": "keyword"
# },
# "path_match": "*"
# }
# }
# ]
# },
# "settings": {
# "index": {
# "provided_name": "my_index",
# "number_of_shards": "1",
# "uuid": "ymMrojPITo-O6b_Huji1yg",
# "number_of_replicas": "1",
# "version": {
# "created": "7010099"
# },
# "creation_date": "1608590601562"
# }
# },
# "aliases": {}
# }
# }

PUT /my_index/_doc/1
{
  "text": "hello world",
  "mykeyword": "hello world again"
}

# 201 Created
# {
# "_type": "_doc",
# "_primary_term": 1,
# "_id": "1",
# "_shards": {
# "successful": 1,
# "total": 2,
# "failed": 0
# },
# "_index": "my_index",
# "result": "created",
# "_version": 1,
# "_seq_no": 0
# }

```

Then I upgraded it to 7.10.1, here's a transcript of the API calls and responses after the upgrade:

```auto
GET /?filter_path=version.number

# 200 OK
# {
# "version": {
# "number": "7.10.1"
# }
# }

GET /my_index

# 200 OK
# {
# "my_index": {
# "mappings": {
# "dynamic_templates": [
# {
# "template_stdField": {
# "mapping": {
# "ignore_malformed": true,
# "type": "keyword"
# },
# "path_match": "*"
# }
# }
# ],
# "properties": {
# "text": {
# "type": "keyword"
# },
# "mykeyword": {
# "type": "keyword"
# }
# }
# },
# "settings": {
# "index": {
# "provided_name": "my_index",
# "number_of_shards": "1",
# "uuid": "ymMrojPITo-O6b_Huji1yg",
# "number_of_replicas": "1",
# "version": {
# "created": "7010099",
# "upgraded": "7100199"
# },
# "creation_date": "1608590601562"
# }
# },
# "aliases": {}
# }
# }

PUT /my_index/_doc/2
{
  "text": "hello world",
  "mykeyword": "hello world again"
}

# 201 Created
# {
# "_type": "_doc",
# "_primary_term": 2,
# "_id": "2",
# "_shards": {
# "successful": 1,
# "total": 2,
# "failed": 0
# },
# "_index": "my_index",
# "result": "created",
# "_version": 1,
# "_seq_no": 1
# }

```

I can't tell what I'm doing differently from you, but I don't get the `mapper_parsing_exception` that you report.

---

<div class="post-metadata">

**Author:** ![DevFullStack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devfullstack/32/77057_2.png) [@DevFullStack](https://discuss.elastic.co/u/DevFullStack)\
**Post date:** [December 21, 2020, 11:39pm UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/8 "2020-12-21T23:39:21Z")

</div>

Oh, I see. I believe since you indexed document `1` with the `mykeyword` property in `v7.1.0`, the record was properly added **AND** the new property `mykeyword` was added to the index mapping. This then allowed you to use it later on the upgraded system.

Now I believe if you try a different dynamically created property on `v7.10.1` when indexing a new document – e.g. `mykeyword2` – you should find the `mapper_parsing_exception `. E.g.:

```auto
//Elasticsearch v7.10.1
PUT /my_index/_doc/3
{
  "text": "hello world",
  "mykeyword2": "hello world again"
}

```

I suppose this issue may not necessarily be unique to upgrade scenarios.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 22, 2020, 10:59am UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/9 "2020-12-22T10:59:22Z")

</div>

Aha ok I see it now. Yes I'd call that a bug; please report it on Github!

---

<div class="post-metadata">

**Author:** ![DevFullStack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/devfullstack/32/77057_2.png) [@DevFullStack](https://discuss.elastic.co/u/DevFullStack)\
**Post date:** [December 22, 2020, 11:13pm UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/10 "2020-12-22T23:13:04Z")

</div>

Done: [https://github.com/elastic/elasticsearch/issues/66765](https://github.com/elastic/elasticsearch/issues/66765)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 19, 2021, 11:13pm UTC](https://discuss.elastic.co/t/ignore-malformed-fires-mapper-parsing-exception-for-keyword-type-on-7-10-1-it-worked-on-7-1-0/259142/11 "2021-01-19T23:13:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.