# ElasticSearch Setup to store data from multiple datacenters **URL:** \ **Category:** Elasticsearch\ **Created:** [November 11, 2013, 10:26am UTC](https://discuss.elastic.co/t/elasticsearch-setup-to-store-data-from-multiple-datacenters/14350 "2013-11-11T10:26:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![dkjhanitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkjhanitt/32/1575_2.png) [@dkjhanitt](https://discuss.elastic.co/u/dkjhanitt)\ **Post date:** [November 11, 2013, 10:26am UTC](https://discuss.elastic.co/t/elasticsearch-setup-to-store-data-from-multiple-datacenters/14350/1 "2013-11-11T10:26:28Z")
Hi, I need to stream data from different datacenters and store them centrally in an elasticsearch cluster. I have to take replication cost and latency into consideration. So in this case, does it makes sense to have elasticsearch data node present in each datacenter and a client sitting outside for query purpose ? If not, can I get suggestion on a good design. Thanks, -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 11, 2013, 10:32am UTC](https://discuss.elastic.co/t/elasticsearch-setup-to-store-data-from-multiple-datacenters/14350/2 "2013-11-11T10:32:07Z")
Cross DC elasticsearch is not a good idea due to latency. You're better off shipping everything to a cluster in one of the DC's. We do this and it works well. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 11 November 2013 21:26, Deepak Jha [dkjhanitt@gmail.com](mailto:dkjhanitt@gmail.com) wrote: > Hi, > I need to stream data from different datacenters and store them centrally > in an elasticsearch cluster. I have to take replication cost and latency > into consideration. So in this case, does it makes sense to have > elasticsearch data node present in each datacenter and a client sitting > outside for query purpose ? If not, can I get suggestion on a good design. > > Thanks, > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dkjhanitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkjhanitt/32/1575_2.png) [@dkjhanitt](https://discuss.elastic.co/u/dkjhanitt)\ **Post date:** [November 11, 2013, 12:54pm UTC](https://discuss.elastic.co/t/elasticsearch-setup-to-store-data-from-multiple-datacenters/14350/3 "2013-11-11T12:54:52Z")
Hi Mark, Thank you for the reply. I am using elasticsearch with Logstash for centralized logging. Having cluster in one DC will have significant network overhead ... Did you do any optimization in this ? If yes, can you please throw some light on it ? On Monday, November 11, 2013 4:02:07 PM UTC+5:30, Mark Walkom wrote: > Cross DC elasticsearch is not a good idea due to latency. > > You're better off shipping everything to a cluster in one of the DC's. We > do this and it works well. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 11 November 2013 21:26, Deepak Jha \<[dkjh...@gmail.com](mailto:dkjh...@gmail.com) \\>wrote: > > > Hi, > > I need to stream data from different datacenters and store them centrally > > in an elasticsearch cluster. I have to take replication cost and latency > > into consideration. So in this case, does it makes sense to have > > elasticsearch data node present in each datacenter and a client sitting > > outside for query purpose ? If not, can I get suggestion on a good design. > > > > Thanks, > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 11, 2013, 9:02pm UTC](https://discuss.elastic.co/t/elasticsearch-setup-to-store-data-from-multiple-datacenters/14350/4 "2013-11-11T21:02:11Z")
Unfortunately, no. It'd be nice if logstash had a compression method during transport but I'm not aware of one. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 11 November 2013 23:54, Deepak Jha [dkjhanitt@gmail.com](mailto:dkjhanitt@gmail.com) wrote: > Hi Mark, > Thank you for the reply. > I am using elasticsearch with Logstash for centralized logging. > Having cluster in one DC will have significant network overhead ... Did > you do any optimization in this ? If yes, can you please throw some light > on it ? > > On Monday, November 11, 2013 4:02:07 PM UTC+5:30, Mark Walkom wrote: > > > Cross DC elasticsearch is not a good idea due to latency. > > > > You're better off shipping everything to a cluster in one of the DC's. We > > do this and it works well. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 11 November 2013 21:26, Deepak Jha [dkjh...@gmail.com](mailto:dkjh...@gmail.com) wrote: > > > > > Hi, > > > I need to stream data from different datacenters and store them > > > centrally in an elasticsearch cluster. I have to take replication cost and > > > latency into consideration. So in this case, does it makes sense to have > > > elasticsearch data node present in each datacenter and a client sitting > > > outside for query purpose ? If not, can I get suggestion on a good design. > > > > > > Thanks, > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dkjhanitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkjhanitt/32/1575_2.png) [@dkjhanitt](https://discuss.elastic.co/u/dkjhanitt)\ **Post date:** [November 12, 2013, 5:30am UTC](https://discuss.elastic.co/t/elasticsearch-setup-to-store-data-from-multiple-datacenters/14350/5 "2013-11-12T05:30:03Z")
Just curious to know what if I have data node in each DC and a client sitting outside to query these.... My query will be mostly DC specific ... Any suggestion ? Thanks, Deepak On Tuesday, November 12, 2013 2:32:11 AM UTC+5:30, Mark Walkom wrote: > Unfortunately, no. > > It'd be nice if logstash had a compression method during transport but I'm > not aware of one. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 11 November 2013 23:54, Deepak Jha \<[dkjh...@gmail.com](mailto:dkjh...@gmail.com) \\>wrote: > > > Hi Mark, > > Thank you for the reply. > > I am using elasticsearch with Logstash for centralized logging. > > Having cluster in one DC will have significant network overhead ... Did > > you do any optimization in this ? If yes, can you please throw some light > > on it ? > > > > On Monday, November 11, 2013 4:02:07 PM UTC+5:30, Mark Walkom wrote: > > > > > Cross DC elasticsearch is not a good idea due to latency. > > > > > > You're better off shipping everything to a cluster in one of the DC's. > > > We do this and it works well. > > > > > > Regards, > > > Mark Walkom > > > > > > Infrastructure Engineer > > > Campaign Monitor > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > On 11 November 2013 21:26, Deepak Jha [dkjh...@gmail.com](mailto:dkjh...@gmail.com) wrote: > > > > > > > Hi, > > > > I need to stream data from different datacenters and store them > > > > centrally in an elasticsearch cluster. I have to take replication cost and > > > > latency into consideration. So in this case, does it makes sense to have > > > > elasticsearch data node present in each datacenter and a client sitting > > > > outside for query purpose ? If not, can I get suggestion on a good design. > > > > > > > > Thanks, > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 12, 2013, 7:44am UTC](https://discuss.elastic.co/t/elasticsearch-setup-to-store-data-from-multiple-datacenters/14350/6 "2013-11-12T07:44:19Z")
You want to avoid having a cross DC cluster, but if you mean two clusters in each DC with a client that reads from both, then yes. That would work as long as you coded the client to take this into account. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 12 November 2013 16:30, Deepak Jha [dkjhanitt@gmail.com](mailto:dkjhanitt@gmail.com) wrote: > Just curious to know what if I have data node in each DC and a client > sitting outside to query these.... My query will be mostly DC specific ... > Any suggestion ? > > Thanks, > Deepak > > On Tuesday, November 12, 2013 2:32:11 AM UTC+5:30, Mark Walkom wrote: > > > Unfortunately, no. > > > > It'd be nice if logstash had a compression method during transport but > > I'm not aware of one. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 11 November 2013 23:54, Deepak Jha [dkjh...@gmail.com](mailto:dkjh...@gmail.com) wrote: > > > > > Hi Mark, > > > Thank you for the reply. > > > I am using elasticsearch with Logstash for centralized logging. > > > Having cluster in one DC will have significant network overhead ... Did > > > you do any optimization in this ? If yes, can you please throw some light > > > on it ? > > > > > > On Monday, November 11, 2013 4:02:07 PM UTC+5:30, Mark Walkom wrote: > > > > > > > Cross DC elasticsearch is not a good idea due to latency. > > > > > > > > You're better off shipping everything to a cluster in one of the DC's. > > > > We do this and it works well. > > > > > > > > Regards, > > > > Mark Walkom > > > > > > > > Infrastructure Engineer > > > > Campaign Monitor > > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > > > On 11 November 2013 21:26, Deepak Jha [dkjh...@gmail.com](mailto:dkjh...@gmail.com) wrote: > > > > > > > > > Hi, > > > > > I need to stream data from different datacenters and store them > > > > > centrally in an elasticsearch cluster. I have to take replication cost and > > > > > latency into consideration. So in this case, does it makes sense to have > > > > > elasticsearch data node present in each datacenter and a client sitting > > > > > outside for query purpose ? If not, can I get suggestion on a good design. > > > > > > > > > > Thanks, > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:07am UTC](https://discuss.elastic.co/t/elasticsearch-setup-to-store-data-from-multiple-datacenters/14350/7 "2017-07-06T02:07:45Z")