How to transfer index from 1 node to a different node

Elastic Stack Elasticsearch
1

Hey guys,

I want to transfer specified indices from source node to a different node (they are not in the same cluster!!!). How can I do this?

Best regards,
PolterFox

2

Use snapshot/restore or reindex from remote.

3

@Christian_Dahlqvist I have used the remote reindex action, but this don´t work correctly.

Here is my command:

POST {
  "source" : {
     "remote":{
        "host" : "hostaddress"
      }, 
      "index": "myremoteindex-2018.12.*"
  }, 
  "dest" : {
    "index": "mylocalindex-2018.12"
  }
}

The problem is that the command don´t "copy" everything from the old indices into the new. Is there anything what I have to do?

Best regards,
PolterFox

4

@elastic can anyone help?

5

What is the error message or the output of the command you ran?

6

@dadoonet the output I receive is just that the command runs into timeout

{
  "statusCode": 504,
  "error": "Gateway Time-out",
  "message":"Client request timeout"
}
7

That's why it does not copy your data I believe.

Can you share your elasticsearch logs?

8

@dadoonet from the elasticsearch server which receives the indice-data or the remote elasticsearch server?

9

@dadoonet This is from the elasticsearch server which should receive the data:

[2019-01-08T08:13:17,430][WARN ][o.e.h.n.Netty4HttpServerTransport] [demo-node] caught exception while handling client http traffic, closing connection [id: 0x21062990, L:/127.0.0.1:9200 - R:/127.0.0.1:55430]
java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert
durch den Hostcomputer abgebrochen
	at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:?]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[?:?]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:?]
	at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1128) ~[netty-buffer-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) ~[netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
10

This is from remote elasticsearch-server:

[2019-01-08T08:06:09,193][INFO ][o.e.m.j.JvmGcMonitorService] [remoteELKCluster] [gc][28915] overhead, spent [342ms] collecting in the last [1s]
[2019-01-08T08:10:55,179][DEBUG][o.e.a.s.TransportSearchScrollAction] [remoteELKCluster] [2156572] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [remoteELKCluster][172.20.136.102:9300][indices:data/read/search[phase/fetch/id/scroll]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@1045082 on QueueResizingEsThreadPoolExecutor[name = remoteELKCluster/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 1ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@55bdcd23[Running, pool size = 4, active threads = 3, queued tasks = 1161, completed tasks = 2827911]]
	at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-6.5.1.jar:6.5.1]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_181]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:98) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor.doExecute(QueueResizingEsThreadPoolExecutor.java:86) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:93) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.search.SearchService.runAsync(SearchService.java:353) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:543) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.action.search.SearchTransportService$10.messageReceived(SearchTransportService.java:396) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.action.search.SearchTransportService$10.messageReceived(SearchTransportService.java:393) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) [x-pack-security-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.1.jar:6.5.1]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:717) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.1.jar:6.5.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
11

Do you have the security module enabled on the remote server?
What exactly did you run? You can send me a private message if you don't want to share the "host" : "hostaddress" part in public.

12

Can you run the following command from the "elasticsearch server which should receive the data"?

curl -XGET hostaddress

Where hostaddress is exactly what you defined in source.remote.host:

POST
{
  "source" : {
     "remote":{
        "host" : "hostaddress"
      }, 
      "index": "myremoteindex-2018.12.*"
  }, 
  "dest" : {
    "index": "mylocalindex-2018.12"
  }
}
13

@dadoonet This is what I get:

{
  "name" : "remoteELKCluster",
  "cluster_name" : "REMOTE_ELK_CLUSTER",
  "cluster_uuid" : "K4wBvAJ0RF6hsCsJNg-_ug",
  "version" : {
    "number" : "6.5.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "8c58350",
    "build_date" : "2018-11-16T02:22:42.182257Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
14

What is the output of the following command on both servers:

GET /_cat/plugins?v
15

both servers just return

name component version
16

Hmmm. I'm running out of ideas.

@Christian_Dahlqvist does this ring a bell?

17

No, not really. Do you have any non-default settings in your Elasticsearch config files (both clusters)?

18

apart from adding each other to the reindex.remote.whitelist, no not really.

19

No security settings? No X-Pack configuration? Nothing else?

20

@Christian_Dahlqvist Cause I have no XPack License, I have nothing changed which sets any security settings. Cause the production server is isolated, but availlable for the second server.