5.3.0 breaks _reindex

[Sebastian_Unger]

Hi There,

I am in the process of slowly migrating our systems from an ancient (and now un-maintained) ES 1.4.1 install to a new 5.x installation. Since I'm doing this in stages, I have currently got 3 ES clusters (each only consisting of a single server):

1. The old production 1.4.1 server ("old")
2. The new production 5.x server ("current")
3. The new test 5.x server ("test")

As part of this process I am using reindex from remote API together with a painless script to get the data from both the old and the current server onto the test server.

Until yesterday (or so) this was all working fairly well. Then both current and test upgraded from 5.2.2 to 5.3.0. Since then the reindex fails. It fails with the following error on reindexing from old to test:

Request:

{
  "dest": {
    "index": "terminalbuildserver",
    "version_type": "internal"
  },
  "source": {
    "index": "terminalbuildserver",
    "_source": true,
    "remote": {
      "host": "http://kibana.terminals.taitradio.net:9200"
    },
    "type": "build"
  },
  "script": {
    "lang": "painless",
    "inline": "\n   if (ctx._source.containsKey('@timestamp'))\n   {\n      ctx._source['timestamp'] = ctx._source.remove('@timestamp')\n   }\n\n   ctx._source['BuildNumber'] = Integer.parseInt(ctx._source.remove('BuildNumber'));\n   ctx._source['Status'] = ctx._source.remove('BuildResult');\n   String oldurl = ctx._source['ChangeLog'];\n   int idx1 = oldurl.lastIndexOf(\"rev=\") + 4;\n   int idx2 = oldurl.lastIndexOf(\"file/\") + 5;\n   ctx._source['ChangeLog'] = oldurl.substring((int)Math.max(idx1, idx2));\n   String cmt = ctx._source['Comment'];\n   if (!cmt.startsWith(\"'\")) Debug.explain(cmt);\n   if (!cmt.endsWith(\"'\")) Debug.explain(cmt);\n   ctx._source['Comment'] = cmt.substring(1, cmt.length() - 1);\n   ctx._source['Artefacts'] = ctx._source.remove('ResultURL');\n   ctx._source['UserEmail'] = ctx._source.remove('Username');\n   ctx._id = ctx._source['Changeset'] + '-' + ctx._source['BuildNumber']\n"
  }
}

Response:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "body={\"error\":\"ElasticsearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 123 in array position 0]; \",\"status\":400}"
      }
    ],
    "type": "status_exception",
    "reason": "body={\"error\":\"ElasticsearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 123 in array position 0]; \",\"status\":400}",
    "caused_by": {
      "type": "response_exception",
      "reason": "POST http://kibana.terminals.taitradio.net:9200/_search/scroll?scroll=5m: HTTP/1.1 400 Bad Request\n{\"error\":\"ElasticsearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 123 in array position 0]; \",\"status\":400}"
    }
  },
  "status": 400
}

Downgrading test back to 5.2.2 restores the previous behaviour.

Is this expected? The docs don't mention any breaking changes between 5.2.2 and 5.3. Anything I can do to get 5.3 working? I don't want to get stuck on another old version.

Cheers,
Seb

[TimV (Tim Vernum)]

Do you have a copy of the server logs for that error?
If you can provide the full stack-trace for it that should help us track it down faster.

Also, can I clarify which servers are running which version in your reindex process.
I take it the reindex is running on 5.3 (indexing into itself).
Which source (remote) server versions are you pulling from? Are they all failing?

Thanks.

[Sebastian_Unger]

There's nothing in the logs. I'll attach them tomorrow to show but they only show the normal stuff.

[Sebastian_Unger]

The reindex is running on a 5.3.0 and is reindexing from a remote 1.4.1.

[jimczi (Jimferenczi)]

Thanks for the report @Sebastian_Unger.
Reindex from remote is broken when the remote cluster is on a version before 2.0.0.
I opened https://github.com/elastic/elasticsearch/pull/23805 to fix this in 5.4.