Upgrading 1.7 -> 5.0 with reindex-from-remote and parent/child docs

Elastic Stack Elasticsearch
1

I'm attempting to move an index from a 1.7 cluster to a 5.0-rc1 cluster via the reindex-from-remote procedure documented here and here.

One type of document reindex works just fine (the parent documents), however a second type which contains a _parent mapping to the first type fails. The reindex task errors with failures of type "routing_missing_exception". (Since the mapping has a _parent field, I believe the _routing is set to 'required'). It appears neither the _parent or _routing fields are being set. After reviewing the documentation I attempted:

  1. Trying all the available settings for the routing option on the dest request e.g. keep/discard/=<some text>

  2. Creating a script to manually assign the _routing field from the _parent field. (Grasping at straws here.) (e.g. "ctx._routing = ctx._parent").

  3. Adding a "fields": "*" to a "query" on the source.

(1) and (2) did nothing, and I received the same error. (3) errored with "Deprecated field [fields] used, expected [stored_fields] instead", but if I then use stored_fields I get "Validation Failed: 1: stored_fields is in not supported in this context;"

The docs say:

By default if _reindex sees a document with routing then the routing is preserved unless it’s changed by the script.

I suspect that is technically true, but _reindex is not seeing the _parent or _routing from the source.

I looked through the source and it appears the reindex-from-remote uses the scan/scroll method, however on 1.7, I have been unable to get a scan/scroll call to return either _parent or _routing in the results.

I even noticed there were tests for parent/child documents from remote in the _reindex source, so I'm thinking I just haven't figured out the right way to do it. What am I missing?

2

Those tests don't cover reindex-from-remote from an old version. I tested a bunch of that stuff manually but I never tested with parent/child. I'm not sure there is a workaround given the rename from fields to stored_fields. I think this highlights the need to have automated testing for the versions we support for reindex from remote. The trouble is that all the fancy code we use to spin up an external cluster for testing needs stuff we added in 5.0. The other backwards compatibility tests all use static files generated by some version of Elasticsearch.

I'll look at this further so I opened

I'll try to provide a workaround in the issue if I find one. At this point I don't expect any fix for this to go into 5.0 so if there isn't a workaround you might be stuck waiting for 5.1 or 5.0.1.

3

I've renamed the issue because I isolated the problem to 1.7. It has the same number but it no longer has the same name so gmail threading issue may occur.

4

@nik9000 Just saw your fix get merged. Thanks! You rock!

1 Like
5

Thanks! I'm backporting to 5.x at the moment. I don't think this'll make 5.0.0 but it'll make 5.0.1.

1 Like
6

As a documentation issue, if the fix isn't going to make it into 5.0.0, is it appropriate to note the issue with parent/child documents and 1.x -> 5.0 reindex-from-remote in the documentation located both here and here?

My worry is there might be a lot of people doing a 1.x to 5.0 upgrade if they had to skip 2.0 due to the "dots in fields" issues (notwithstanding 2.4). Right now the docs callout reindex-from-remote as the only way to do that short of an intervening migration to 2.4. I was thinking it might be a good idea to present a warning regarding the parent/child issue if it is going to there at 5.0 release (or perhaps also suggest people with that issue wait until 5.0.1).

I'm happy to submit a PR to the docs if you think it is a good idea.

7

I would merge it with glee.

And, yes, I expect to merge the fix for 5.0.1.

8

Not sure if I targeted the right branch, but FWIW: