Sync local Elasticsearch index with remote Elasticsearch continuously

Elastic Stack Elasticsearch
1

Hi All,
I have local Elasticsearch that collects a lot of data into the index lets call it eventIndex.
My goal is to move this data from "localEventIndex" to the remote Elasticsearch "remoteEventIndex" index.
I made some POC with Logstash that sends the data to the remote Elasticsearch index and marking it as "uploaded" for avoiding additional upload of same document. It works OK , but if there will be some connection issues, some documents may be sent twice or more, in other words there is no 100% data accuracy insurance in this case.
I'm wondering maybe there are some other nice solutions that I can use in order to move data from local Index to remote index with high accuracy.
Thanks in advance.

2

That is exactly what cross-cluster replication was created for. It does require a commercial license though. Before this was available a common approach was to use a message queue and have two consumers writing to different clusters.

3

Thanks for the answer @Christian_Dahlqvist

If I'm not wrong " [cross-cluster replication] https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-ccr.html)" works on "PULL" mechanism from the remote on a local server, in my case "PUSH" mechanism from local to remote is preferred, also as open-source.
maybe there are other solutions ?(beside message ques with 2 writing consumers)

Thanks

4

I am not aware of any open-source solution.

5

I don't understand the distinction you're making between "PULL" and "PUSH" mechanisms, or at least I don't understand why it's meaningful to you. Can you explain why CCR's mechanism doesn't work for you? Although it's technically a pull-based mechanism it uses long-polling so it's functionally much more like a push.

6

@DavidTurner Thanks
The thing is that local ES is at customer site, when remote ES is on Cloud.
We can’t open the customer site for to allow HTTP requests from Cloud. So we need to run Logstash or something else on the customer side. To query data from the local ES and to push to the remote ES.
Thanks

7

I see, thanks. This is surmountable with a bit of network shenanigans (a VPN or some other kind of tunnel) but I can see how it's a bit of an obstacle.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.