Migrating indices from one machine to another

SamehSaeed · October 19, 2024, 7:28pm

Hello, I'm trying to migrate indices from a windows machine to a linux machine. For security reasons, I cannot use anything related to cloud.
Is there a way to migrate indices using local file system? I tried copying the snapshot to the new machine but it did not work.

rugenl · October 20, 2024, 10:26pm

Remote remindex?

SamehSaeed · October 28, 2024, 10:43am

I'm trying to add a remote cluster though I cant get it to work, I tried adding the linux server in under the remote cluster option but it says "establish trust" which I don't know what it means, if I skip this and add the server it wont work.

Error :

[2024-10-28T13:15:47,626][WARN ][r.suppressed             ] [SCPRWUWS05] path: /*%3A.monitoring-es-*%2C.monitoring-es-*%2C*%3Ametrics-elasticsearch.stack_monitoring.cluster_stats-*%2Cmetrics-elasticsearch.stack_monitoring.cluster_stats-*/_search, params: {index=*:.monitoring-es-*,.monitoring-es-*,*:metrics-elasticsearch.stack_monitoring.cluster_stats-*,metrics-elasticsearch.stack_monitoring.cluster_stats-*, filter_path=hits.hits._source.cluster_settings.cluster.metadata.display_name,hits.hits._source.cluster_uuid,hits.hits._source.elasticsearch.cluster.id,hits.hits._source.cluster_name,hits.hits._source.elasticsearch.cluster.name}, status: 500
org.elasticsearch.transport.RemoteTransportException: [error while communicating with remote cluster [Linux_Cluster]]
Caused by: org.elasticsearch.transport.ConnectTransportException: [][10.21.144.85:443] general node connection failure
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.lambda$onResponse$2(TcpTransport.java:1132) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:179) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleLocalException(TransportHandshaker.java:263) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$0(TransportHandshaker.java:154) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:131) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:315) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:229) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.transport.netty4.Netty4Utils.lambda$addListener$2(Netty4Utils.java:173) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[?:?]
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[?:?]
	at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1164) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:755) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:731) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:620) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.transport.TransportException: handshake failed because connection reset
	at org.elasticsearch.transport.TransportHandshaker.lambda$sendHandshake$0(TransportHandshaker.java:154) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:131) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:366) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:286) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:315) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:229) ~[elasticsearch-8.13.2.jar:?]
	at org.elasticsearch.transport.netty4.Netty4Utils.lambda$addListener$2(Netty4Utils.java:173) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[?:?]
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[?:?]
	at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1164) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:755) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:731) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:620) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]

rugenl · October 28, 2024, 1:04pm

It's been awhile since I did this, and I don't have a place to test, but I don't think reindex needs a remote cluster definition. See Reindex API | Elasticsearch Guide [8.15] | Elastic

But, it may need "good" ssl certs, self-signed may not work. Certs is probably your "establish trust" issue above.

SamehSaeed · October 30, 2024, 6:32am

Thnx alot for the info, that really helped!

I read in this post how to solve the SSL handshake exception, after adding the "source host" certificate to the "destination host" jvm cacerts file, the process ran without any exceptions.

Just a quick question, is there a way to run this process once to migrate all indices? we have around 400

rugenl · October 30, 2024, 2:11pm

This might work, but I've never used it Bash script to re-index all indices matching apattern from a remote Elasticsearch cluster to a local one · GitHub

Topic		Replies	Views
How to transfer index from 1 node to a different node Elasticsearch	26	1185	February 5, 2019
How to transfer elastic index created in local machine to elastic cloud? Elastic Cloud Enterprise (ECE)	4	1204	July 26, 2018
Index migration from one elasticsearch cluster to another Elasticsearch	4	117	March 28, 2024
Reindex to remote server Elasticsearch	7	4126	September 14, 2017
How to use transport client (or something else) to search in ES indices on another system? Elasticsearch	9	1425	July 5, 2017

Migrating indices from one machine to another

Related topics