How to remove duplicate fields within doc?

Elastic Stack Elasticsearch
1

Hi,

Trying to migrate from 5.X to 6.X but have issues when reading ES data that was stored before the migration. The error I'm getting back is:

{error,{400,
        [{<<"error">>,
          [{<<"root_cause">>,
            [[{<<"type">>,<<"parse_exception">>},
              {<<"reason">>,<<"Failed to parse content to map">>}]]},
           {<<"type">>,<<"search_phase_execution_exception">>},
           {<<"reason">>,<<"all shards failed">>},
           {<<"phase">>,<<"fetch">>},
           {<<"grouped">>,true},
           {<<"failed_shards">>,
            [[{<<"shard">>,0},
              {<<"index">>,<<"MY_INDEX">>},
              {<<"node">>,<<"MY_NODE">>},
              {<<"reason">>,
               [{<<"type">>,<<"parse_exception">>},
                {<<"reason">>,<<"Failed to parse content to map">>},
                {<<"caused_by">>,
                 [{<<"type">>,<<"json_parse_exception">>},
                  {<<"reason">>,
                   <<"Duplicate field 'X_ID'\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@28e05854; line: 1, column: 2061]">>}]}]}]]}]},
         {<<"status">>,400}]}}

I've resolved an issue in source code where duplicate fields (MY_ID) were being entered. But now that I have documents with duplicate fields, is there a way to remove those fields? Otherwise, I cannot read any documents that were added on 5.X.

2

Hi, you could try setting an _source.include parameter when reading data from ES 5.x.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.