# How to remove duplicate fields within doc?

**URL:** <https://discuss.elastic.co/t/how-to-remove-duplicate-fields-within-doc/254316>\
**Category:** Elasticsearch\
**Created:** [November 4, 2020, 6:08pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-fields-within-doc/254316 "2020-11-04T18:08:47Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kyle\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kyle_neal/32/45151_2.png) [@Kyle\_Neal](https://discuss.elastic.co/u/Kyle_Neal)\
**Post date:** [November 4, 2020, 6:08pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-fields-within-doc/254316/1 "2020-11-04T18:08:47Z")

</div>

Hi,

Trying to migrate from 5.X to 6.X but have issues when reading ES data that was stored before the migration. The error I'm getting back is:

```auto
{error,{400,
        [{<<"error">>,
          [{<<"root_cause">>,
            [[{<<"type">>,<<"parse_exception">>},
              {<<"reason">>,<<"Failed to parse content to map">>}]]},
           {<<"type">>,<<"search_phase_execution_exception">>},
           {<<"reason">>,<<"all shards failed">>},
           {<<"phase">>,<<"fetch">>},
           {<<"grouped">>,true},
           {<<"failed_shards">>,
            [[{<<"shard">>,0},
              {<<"index">>,<<"MY_INDEX">>},
              {<<"node">>,<<"MY_NODE">>},
              {<<"reason">>,
               [{<<"type">>,<<"parse_exception">>},
                {<<"reason">>,<<"Failed to parse content to map">>},
                {<<"caused_by">>,
                 [{<<"type">>,<<"json_parse_exception">>},
                  {<<"reason">>,
                   <<"Duplicate field 'X_ID'\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@28e05854; line: 1, column: 2061]">>}]}]}]]}]},
         {<<"status">>,400}]}}

```

I've resolved an issue in source code where duplicate fields (`MY_ID`) were being entered. But now that I have documents with duplicate fields, is there a way to remove those fields? Otherwise, I cannot read any documents that were added on 5.X.

---

<div class="post-metadata">

**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\
**Post date:** [November 15, 2020, 7:08am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-fields-within-doc/254316/2 "2020-11-15T07:08:02Z")

</div>

Hi, you could try setting an \_source.include parameter when reading data from ES 5.x.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2020, 7:08am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-fields-within-doc/254316/3 "2020-12-13T07:08:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.