# Reindex from Remote not working **URL:** \ **Category:** Elasticsearch\ **Created:** [December 20, 2016, 11:52pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625 "2016-12-20T23:52:13Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 20, 2016, 11:52pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/1 "2016-12-20T23:52:13Z")
I'm investigating upgrading from our ES 1.7.5 system (running on Windows) to ES 5.1.1. I'm doing some testing with _ **Reindex from Remote** _ to see if I can simply bring my current cluster's data over to a 5.x system. However, it's failing for me. When I issue the command as shown here [Reindex From Remote](https://www.elastic.co/guide/en/elasticsearch/reference/5.x/docs-reindex.html#reindex-from-remote), I receive the following error: ```auto { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Cannot guess the xcontent type without mark/reset support on class org.apache.http.nio.entity.ContentInputStream" } ], "type": "illegal_argument_exception", "reason": "Cannot guess the xcontent type without mark/reset support on class org.apache.http.nio.entity.ContentInputStream" }, "status": 400 } ``` To get more info, I decided to up the `rootLogger.level` to debug. When I run the request again, I get a different error: ```auto { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Required [version]" } ], "type": "illegal_argument_exception", "reason": "Required [version]" }, "status": 400 } ``` The debug logging I've seen doesn't really seem to be posting a request to the remote system. All I've seen in the logs are GETs like: ```auto [2016-12-20T16:04:38,708][DEBUG][o.e.c.RestClient] request [GET http://myremoteserver:9200/] returned [HTTP/1.1 200 OK] [2016-12-20T16:04:38,714][DEBUG][tracer] curl -iX GET 'http://myremoteserver:9200/' # HTTP/1.1 200 OK # Content-Type: text/plain; charset=UTF-8 # Content-Length: 9 # # {"OK":{}} ``` Is there some incompatibility between 1.7.5 and 5.1.1? Is this some other known issue? Thanks, Tom Doman [tom@degreed.com](mailto:tom@degreed.com) ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 21, 2016, 12:09am UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/2 "2016-12-21T00:09:21Z")
> [@Thomas\_Doman](#): > > Is there some incompatibility between 1.7.5 and 5.1.1? Is this some other known issue? No, it isn't. Though I recently fixed another thing to do with content detection and `Cannot guess the xcontent type without mark/reset support on class org.apache.http.nio.entity.ContentInputStream` seems likely to be a bug. Usually the content type detection isn't needed.... What do you get back when you `curl -iX GET 'http://myremoteserver:9200/'`? The response that the tracer logged doesn't look like the kind of response I'd expect from Elasticsearch. ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 21, 2016, 6:21pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/3 "2016-12-21T18:21:37Z")
Yeah, what I showed above is pretty much the extent of it. It looks like it never really issues the query to get data from the remote server at all so there's no info to use to guess at the content type ... there **_is_** no content. Note that the log snippet below is what happens when I have the log level turned up to **debug** which doesn't yield the `Cannot guess the xcontent type` exception. Anyway, here's a little more from the log in case it helps: ```auto [2016-12-21T11:16:57,023][DEBUG][o.a.h.i.n.c.PoolingNHttpClientConnectionManager] Connection released: [id: http-outgoing-0][route: {}->http://myremoteserver:9200][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 30] [2016-12-21T11:16:57,025][DEBUG][o.e.c.RestClient] request [GET http://myremoteserver:9200/] returned [HTTP/1.1 200 OK] [2016-12-21T11:16:57,028][DEBUG][tracer] curl -iX GET 'http://myremoteserver:9200/' # HTTP/1.1 200 OK # Content-Type: text/plain; charset=UTF-8 # Content-Length: 9 # # {"OK":{}} [2016-12-21T11:16:57,030][DEBUG][o.a.h.i.n.c.PoolingNHttpClientConnectionManager] Connection manager is shutting down [2016-12-21T11:16:57,030][DEBUG][o.a.h.i.n.c.ManagedNHttpClientConnectionImpl] http-outgoing-0 10.0.0.253:35888<->13.85.73.78:9200[ACTIVE][r:r]: Close [2016-12-21T11:16:57,532][DEBUG][o.a.h.i.n.c.InternalIODispatch] http-outgoing-0 [CLOSED]: Disconnected [2016-12-21T11:16:58,032][DEBUG][o.a.h.i.n.c.PoolingNHttpClientConnectionManager] Connection manager shut down [2016-12-21T11:16:58,038][DEBUG][r.suppressed] path: /_reindex, params: {} java.lang.IllegalArgumentException: Required [version] at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.finish(ConstructingObjectParser.java:324) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.common.xcontent.ConstructingObjectParser$Target.access$000(ConstructingObjectParser.java:237) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:143) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:77) ~[elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.index.reindex.remote.RemoteScrollableHitSource$1RetryHelper$1.onSuccess(RemoteScrollableHitSource.java:164) [reindex-5.1.1.jar:5.1.1] at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:554) [rest-5.1.1.jar:5.1.1] at org.elasticsearch.client.RestClient$1.completed(RestClient.java:309) [rest-5.1.1.jar:5.1.1] at org.elasticsearch.client.RestClient$1.completed(RestClient.java:300) [rest-5.1.1.jar:5.1.1] at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) [httpcore-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) [httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.client.InternalRequestExecutor.inputReady(InternalRequestExecutor.java:83) [httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) [httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) [httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [httpcore-nio-4.4.5.jar:4.4.5] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111] [2016-12-21T11:16:58,076][DEBUG][o.a.h.i.n.c.InternalIODispatch] http-outgoing-0 [CLOSED] [content length: 9; pos: 9; completed: true] [2016-12-21T11:16:58,084][DEBUG][i.n.h.c.c.ZlibCodecFactory] -Dio.netty.noJdkZlibDecoder: false [2016-12-21T11:16:58,085][DEBUG][i.n.h.c.c.ZlibCodecFactory] -Dio.netty.noJdkZlibEncoder: false ``` ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 21, 2016, 8:38pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/4 "2016-12-21T20:38:34Z")
Is there a stack trace under that? ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 21, 2016, 8:58pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/5 "2016-12-21T20:58:21Z")
Yep, sorry I should have included that one originally. I just added it to the log section above. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 21, 2016, 9:01pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/6 "2016-12-21T21:01:57Z")
OK! So you've uncovered some bugs in reindex-from-remote, but I think they are in the error-reporting portion. You should be able to work around the issues. It **looks** like you are pointing to something that isn't responding with Elaticsearch's normal JSON. That is why I asked what it looked like if you manually executed those curl commands. If they output what the tracer logged then reindex-from-remote isn't going to work because the thing on the other side doesn't look like Elasticsearch. ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 21, 2016, 9:26pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/7 "2016-12-21T21:26:03Z")
Hmmm, if I use the Sense plugin for Chrome and send the following request to my remote 1.7.5 server, it responds w/ data, lots of it. ```auto POST articles/article/_search { "query": { "match_all": {} } } ``` But if I post this to my 5.1.1 system, I get the error we've been discussing: ```auto POST _reindex { "source": { "remote": { "host": "http://myremoteserver:9200", "username": "user", "password": "pwd" }, "index": "articles", "query": { "match_all": {} } }, "dest": { "index": "articles_dest" } } ``` It really doesn't seem to matter what I put in the `query`, it always gives the same error which makes me think that query request isn't being made. Based on the log, I don't see that in the trace. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 21, 2016, 9:28pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/8 "2016-12-21T21:28:02Z")
And what happens if you shell into the 5.1 machine and run exactly ```auto curl -iX GET 'http://myremoteserver:9200/' ``` ? ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 21, 2016, 9:36pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/9 "2016-12-21T21:36:48Z")
No need to shell into the 5.1 machine, it's my local Windows 10 box. 🙂 Here's what I get when I use curl: ```auto HTTP/1.1 200 OK Content-Type: text/plain; charset=UTF-8 Content-Length: 9 {"OK":{}} ``` Which is exactly what we see in the logs. ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 21, 2016, 9:40pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/10 "2016-12-21T21:40:40Z")
If I use the following, I get a nice response: ```auto curl -iX GET "http://user:pwd@myremoteserver:9200/_search" ``` Note: it's windows, needs the double quotes. 🙂 ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 21, 2016, 10:00pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/11 "2016-12-21T22:00:07Z")
OK. I think the trouble is that Elasticsearch isn't use preemptive authorization, instead sending the request and retrying if it is told that it needs authorization. Can you make whatever thing is doing the password checking return a `403 Forbidden`? Maybe reindex from remote should do preemptive authorization to work around situations like this.... ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 21, 2016, 10:35pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/12 "2016-12-21T22:35:47Z")
Isn't the "thing doing the password checking" ES 1.7.5? I expected 5.1 would do preemptive authorization since I've provided it with a user name and password. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 22, 2016, 12:13am UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/13 "2016-12-22T00:13:58Z")
> [@Thomas\_Doman](#): > > Isn't the "thing doing the password checking" ES 1.7.5? Is it Shield? > [@Thomas\_Doman](#): > > I expected 5.1 would do preemptive authorization since I've provided it with a user name and password. Yeah, I expected that is what it'd do when I first implemented it but it isn't the default in httpclient most things send the right response. We test reindex-from-remote from modern versions of shield but not 1.7. ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 22, 2016, 12:45am UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/14 "2016-12-22T00:45:49Z")
Nope, we do not use shield in any of our environments. In all of the above, we're going directly to ES. For our non-sensitive dev environments, we just have basic auth set up. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 22, 2016, 1:47am UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/15 "2016-12-22T01:47:55Z")
> [@Thomas\_Doman](#): > > we just have basic auth set up. That isn't a feature built into Elasticsearch though. ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 22, 2016, 3:20pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/16 "2016-12-22T15:20:32Z")
My 1.7.5 `elasticsearch.yml` file has the following configuration: ```auto http.basic.enabled: true http.basic.user: "user" http.basic.password: "pwd" ``` If that's not "built in", I don't know what is. As I showed above, invoking a search via CURL w/ my configured user name and password works properly. When I use Sense, the browser prompts me for the user name and password the first time. This feels like a bug to me, I assumed user error at first but sounds like it may be worthwhile to raise it as an issue. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 22, 2016, 4:07pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/17 "2016-12-22T16:07:43Z")
> [@Thomas\_Doman](#): > > This feels like a bug to me, I assumed user error at first but sounds like it may be worthwhile to raise it as an issue. Yeah, it is a bug. I'm trying to help you work around it because I'm not going to get a chance to fix it for a bit. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 22, 2016, 4:16pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/18 "2016-12-22T16:16:24Z")
I've files these issues: > > > All of which you found. ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [December 22, 2016, 4:22pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/19 "2016-12-22T16:22:54Z")
Perfect, thank you very much Nik, I appreciate it! I'm looking at all my options to get from 1.x to 5.x and if reindex from remote works, that looks like it'll be the best option for us to get there. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 22, 2016, 4:28pm UTC](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625/20 "2016-12-22T16:28:56Z")
It looks like [this](https://github.com/Asquera/elasticsearch-http-basic) plugin might be how you got basic auth. Do you know if this is it? [Next page](https://discuss.elastic.co/t/reindex-from-remote-not-working/69625.md?page=2)