Reindex 5.4.0 to 7.1.1

[Nick_Carn]

Hi there,

I'm reindexing from remote. I have two slightly different document types (not es types). One i can reindex without problems, the other i get the error below. I can create a document manually from the _source but not using the reindex api.

Any ideas?

Thanks,
Nick

https://pastebin.com/U3vgG5ST

  "error" : {
    "type" : "exception",
    "reason" : "Error parsing the response, remote is likely not an Elasticsearch instance",
    "caused_by" : {
      "type" : "x_content_parse_exception",
      "reason" : "[1:1437] [search_response] failed to parse field [hits]",
      "caused_by" : {
        "type" : "x_content_parse_exception",
        "reason" : "[1:1437] [hits] failed to parse field [hits]",
        "caused_by" : {
          "type" : "x_content_parse_exception",
          "reason" : "[1:1437] [hit] failed to parse field [_source]",
          "caused_by" : {
            "type" : "parsing_exception",
            "reason" : "[hit] failed to parse [_source]",
            "line" : 1,
            "col" : 1437,
            "caused_by" : {
              "type" : "json_parse_exception",
              "reason" : "Duplicate field 'locationType'\n at [Source: org.apache.http.nio.entity.ContentInputStream@7f406601; line: 1, column: 1464]",
              "suppressed" : [
                {
                  "type" : "illegal_state_exception",
                  "reason" : "Failed to close the XContentBuilder",
                  "caused_by" : {
                    "type" : "i_o_exception",
                    "reason" : "Unclosed object or array found"
                  }
                }
              ]
            }

[wangqinghuan (Wang)]

The problem is "duplicate field", you are indexing documents with duplicate fields into a newer version elasticsearch that does not allow duplicate fields to exist by default.

[Nick_Carn]

HI Wang,

Thank you for looking and replying.

There is definitely no duplicate field. I tried to reindex just one of the documents that fail and it throws that i_o_exception. I can index that document manually by posting the json, just not through the reindex API.

Here's the soure document https://pastebin.com/KtHTezMw

I even tried to index the document manually to create the dynamic mappings, deleting the doc and then reindex it through the API, and that fails with the same exception.

Any help appreciated.

Thanks,
Nick

[Nick_Carn]

I figured it out. I had to remove the _type, but that still threw the exception. So i also had to be explicit about what i wanted to include.

POST _reindex?wait_for_completion=false
{
  "source": {
"remote": {
  "host": "http://training-es5.videotel-stage.com:9200"
},
"index": "cts-training",
"_source": {
  "include": [
    "key",
    "origin",
    "originKey",
    "generationDate",
    "who",
    "when",
    "where",
    "what",
    "verified"
    ]
},      
  "query": { "bool": {"must_not": [
{ "term": {
  "type": {
    "value": "training"
  }
}}
  ]}}
  },
  "dest": {
"index": "cts-training2"
  },
  "script": {
"source": """
  ctx._source.type = ctx._type;
  ctx._type = '_doc';
"""
  }
 }