Elasticsearch in a federated mode


(ajan) #1

I have a requirement to analyze on how elasticsearch be used in
federation.

Let me try to explain;

Lets say, as a part of a big enterprise, I have a ES cluster - ES001
with 3 nodes, using the default configurations of 1 replica and 2
shards deployed in LA, USA. Another ES cluster - ES002 with the same
configuration as ES001 deployed at Cork, Ireland.

Is there a way to synchronize between these 2 clusters? I'm not
talking about data replication, can a search query on cluster ES001 be
directed to ES002 based on some parameter?

With best regards,
Afzal Jan


(Shay Banon) #2

?. You code issues the search request, you can decide which one to hit.

On Thursday, March 1, 2012 at 9:02 AM, ajan wrote:

I have a requirement to analyze on how elasticsearch be used in
federation.

Let me try to explain;

Lets say, as a part of a big enterprise, I have a ES cluster - ES001
with 3 nodes, using the default configurations of 1 replica and 2
shards deployed in LA, USA. Another ES cluster - ES002 with the same
configuration as ES001 deployed at Cork, Ireland.

Is there a way to synchronize between these 2 clusters? I'm not
talking about data replication, can a search query on cluster ES001 be
directed to ES002 based on some parameter?

With best regards,
Afzal Jan


(Radu Gheorghe) #3

I suppose you want to run queries as if it was one cluster, but insert
data only on the local nodes, and not migrate data between different
locations.

If that's the case, you could also search both clusters and aggregate
the results in your app. Or, you can put all nodes in the same ES
cluster, and use shard allocation to make sure local data gets indexed
in local nodes.

In case you find out other options, I'd be really interested :smiley:

On Mar 1, 2:35 pm, Shay Banon kim...@gmail.com wrote:

?. You code issues the search request, you can decide which one to hit.

On Thursday, March 1, 2012 at 9:02 AM, ajan wrote:

I have a requirement to analyze on how elasticsearch be used in
federation.

Let me try to explain;

Lets say, as a part of a big enterprise, I have a ES cluster - ES001
with 3 nodes, using the default configurations of 1 replica and 2
shards deployed in LA, USA. Another ES cluster - ES002 with the same
configuration as ES001 deployed at Cork, Ireland.

Is there a way to synchronize between these 2 clusters? I'm not
talking about data replication, can a search query on cluster ES001 be
directed to ES002 based on some parameter?

With best regards,
Afzal Jan


(Radu Gheorghe) #4

On Mar 2, 4:17 pm, Radu Gheorghe radu0gheor...@gmail.com wrote:

I suppose you want to run queries as if it was one cluster, but insert
data only on the local nodes, and not migrate data between different
locations.

If that's the case, you could also search both clusters and aggregate
the results in your app. Or, you can put all nodes in the same ES
cluster, and use shard allocation to make sure local data gets indexed
in local nodes.

Sorry, that would be "Shard Allocation Filtering". See here:
http://www.elasticsearch.org/guide/reference/modules/cluster.html

In case you find out other options, I'd be really interested :smiley:

On Mar 1, 2:35 pm, Shay Banon kim...@gmail.com wrote:

?. You code issues the search request, you can decide which one to hit.

On Thursday, March 1, 2012 at 9:02 AM, ajan wrote:

I have a requirement to analyze on how elasticsearch be used in
federation.

Let me try to explain;

Lets say, as a part of a big enterprise, I have a ES cluster - ES001
with 3 nodes, using the default configurations of 1 replica and 2
shards deployed in LA, USA. Another ES cluster - ES002 with the same
configuration as ES001 deployed at Cork, Ireland.

Is there a way to synchronize between these 2 clusters? I'm not
talking about data replication, can a search query on cluster ES001 be
directed to ES002 based on some parameter?

With best regards,
Afzal Jan


(system) #5