# Deploy develpment, test and live instances **URL:** \ **Category:** Elasticsearch\ **Created:** [September 11, 2013, 11:32am UTC](https://discuss.elastic.co/t/deploy-develpment-test-and-live-instances/13553 "2013-09-11T11:32:41Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![alexanderjmitchell](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexanderjmitchell/32/1680_2.png) [@alexanderjmitchell](https://discuss.elastic.co/u/alexanderjmitchell)\ **Post date:** [September 11, 2013, 11:32am UTC](https://discuss.elastic.co/t/deploy-develpment-test-and-live-instances/13553/1 "2013-09-11T11:32:41Z")
We are designing our first elasticsearch deployment. We need a development 'instance', test 'instance' and live 'instance'. I use the word 'instance' as I'm not sure at this stage whether it should be a node or cluster. Two essential requirements for us are: 1. No cross talk between any 'instance' (ie no queries for the live 'instance' being routed to the test or development) 2. The ability to synchronise data from the development 'instance' to the test and then live 'instances' on demand, not automatically. ie, we want to develop against a particular dataset, push that data to test, conduct formal testing and then push that data to our live instance if and only if the tests pass. We are evaluating two options: 1. One cluster, three nodes, shards replicated across nodes: If we were to have one cluster, with three nodes, I think we could prevent cross talk using routing (and possibly preference?) but I don't see a way of preventing elasticsearch from synchronising data from the configured master node (on development) to the test and live shards if they are part of the same cluster. I suppose we could remove test and live from the cluster and then rejoin them to carry out the updates? Is there a more natural way that we can achieve this 'update on demand'? Am I correct that we can prevent cross talk between the nodes through configuration? 1. Three clusters, each with one node: It seems that some people ([http://elasticsearch-users.115913.n3.nabble.com/Replicating-data-from-one-ES-cluster-to-another-td4034172.html](http://elasticsearch-users.115913.n3.nabble.com/Replicating-data-from-one-ES-cluster-to-another-td4034172.html)) use rsync ([http://rsync.samba.org/](http://rsync.samba.org/)) to update the underlying Lucene index files between clusters once an update is complete, so we could have development, test and live clusters and use rsync to push data between them as required. I've worked at a relatively low level with Lucene before and am broadly happy that this approach would work, but can anyone provide any experience of trying it? I've searched for an elasticsearch-to-elasticsearch river but have not been able to find one. If one exists that would obviously be a good option. Does anyone know of one? Broadly, I can't beleive that our usage pattern of separated dev -\> test -\> live with on demand updates has not been used elsewhere, and that this problem hasn't been solved before. Does anyone have any experience of solving this? Have we missed something obvious here? Thanks Alex -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 11, 2013, 12:53pm UTC](https://discuss.elastic.co/t/deploy-develpment-test-and-live-instances/13553/2 "2013-09-11T12:53:42Z")
Hi Alex, I think you could have both: 1. Create one index per platform (I mean dev, test, live) 2. Have 3 clusters. For the 1st one, the main problem I see if when it comes to upgrading a cluster (elasticsearch version). I'm not sure you want to test it in production, right? For the 2nd one, which is the one I prefer, I'd only set one node for dev, one or 3 for test, 3 (at least) for production. Unless you don't care of having downtime. ES River was something I had in mind a long time ago: [[Feature Request] Add a river to ElasticSearch instance · Issue #1077 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1077) But it was not really making sense. We are currently working on Snapshot & restore features which could help you a lot to do what you are looking for. That said, it will only work if you have exactly same data on all platforms. Does it help or did I miss something? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 11 sept. 2013 à 13:32, Alexander Mitchell [alexanderjmitchell@gmail.com](mailto:alexanderjmitchell@gmail.com) a écrit : > We are designing our first elasticsearch deployment. We need a development 'instance', test 'instance' and live 'instance'. I use the word 'instance' as I'm not sure at this stage whether it should be a node or cluster. > > Two essential requirements for us are: > > 1. No cross talk between any 'instance' (ie no queries for the live 'instance' being routed to the test or development) > 2. The ability to synchronise data from the development 'instance' to the test and then live 'instances' on demand, not automatically. ie, we want to develop against a particular dataset, push that data to test, conduct formal testing and then push that data to our live instance if and only if the tests pass. > > We are evaluating two options: > > 1. One cluster, three nodes, shards replicated across nodes: > > If we were to have one cluster, with three nodes, I think we could prevent cross talk using routing (and possibly preference?) but I don't see a way of preventing elasticsearch from synchronising data from the configured master node (on development) to the test and live shards if they are part of the same cluster. I suppose we could remove test and live from the cluster and then rejoin them to carry out the updates? Is there a more natural way that we can achieve this 'update on demand'? Am I correct that we can prevent cross talk between the nodes through configuration? > > 1. Three clusters, each with one node: > > It seems that some people ([http://elasticsearch-users.115913.n3.nabble.com/Replicating-data-from-one-ES-cluster-to-another-td4034172.html](http://elasticsearch-users.115913.n3.nabble.com/Replicating-data-from-one-ES-cluster-to-another-td4034172.html)) use rsync ([http://rsync.samba.org/](http://rsync.samba.org/)) to update the underlying Lucene index files between clusters once an update is complete, so we could have development, test and live clusters and use rsync to push data between them as required. I've worked at a relatively low level with Lucene before and am broadly happy that this approach would work, but can anyone provide any experience of trying it? > > I've searched for an elasticsearch-to-elasticsearch river but have not been able to find one. If one exists that would obviously be a good option. Does anyone know of one? > > Broadly, I can't beleive that our usage pattern of separated dev -\> test -\> live with on demand updates has not been used elsewhere, and that this problem hasn't been solved before. Does anyone have any experience of solving this? Have we missed something obvious here? > > Thanks > > Alex > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![alexanderjmitchell](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexanderjmitchell/32/1680_2.png) [@alexanderjmitchell](https://discuss.elastic.co/u/alexanderjmitchell)\ **Post date:** [September 11, 2013, 3:40pm UTC](https://discuss.elastic.co/t/deploy-develpment-test-and-live-instances/13553/3 "2013-09-11T15:40:34Z")
Hi David Thanks for your response. I'm not considering the need for replication within an 'instance' at the moment as failover is not a high priority requirement. Regarding option 1, is there a natural way that we can achieve the 'update on demand' that we need? Am I correct that we can prevent cross talk between the nodes through configuration? Do you have any experience of rsync? Thanks again Alex On Wednesday, 11 September 2013 13:53:42 UTC+1, David Pilato wrote: > Hi Alex, > > I think you could have both: > > 1. Create one index per platform (I mean dev, test, live) > 2. Have 3 clusters. > > For the 1st one, the main problem I see if when it comes to upgrading a > cluster (elasticsearch version). > I'm not sure you want to test it in production, right? > > For the 2nd one, which is the one I prefer, I'd only set one node for dev, > one or 3 for test, 3 (at least) for production. > Unless you don't care of having downtime. > > ES River was something I had in mind a long time ago: > [[Feature Request] Add a river to ElasticSearch instance · Issue #1077 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1077) > But it was not really making sense. > > We are currently working on Snapshot & restore features which could help > you a lot to do what you are looking for. > That said, it will only work if you have exactly same data on all > platforms. > > Does it help or did I miss something? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 11 sept. 2013 à 13:32, Alexander Mitchell \<[alexander...@gmail.com](mailto:alexander...@gmail.com)\\> > a écrit : > > We are designing our first elasticsearch deployment. We need a development > 'instance', test 'instance' and live 'instance'. I use the word 'instance' > as I'm not sure at this stage whether it should be a node or cluster. > > Two essential requirements for us are: > > 1. No cross talk between any 'instance' (ie no queries for the live > 'instance' being routed to the test or development) > 2. The ability to synchronise data from the development 'instance' to the > test and then live 'instances' on demand, not automatically. ie, we want to > develop against a particular dataset, push that data to test, conduct > formal testing and then push that data to our live instance if and only if > the tests pass. > > We are evaluating two options: > > 1. One cluster, three nodes, shards replicated across nodes: > > If we were to have one cluster, with three nodes, I think we could prevent > cross talk using routing (and possibly preference?) but I don't see a way > of preventing elasticsearch from synchronising data from > the configured master node (on development) to the test and live shards if > they are part of the same cluster. I suppose we could remove test and live > from the cluster and then rejoin them to carry out the updates? Is there a > more natural way that we can achieve this 'update on demand'? Am I correct > that we can prevent cross talk between the nodes through configuration? > > 1. Three clusters, each with one node: > > It seems that some people ( > [http://elasticsearch-users.115913.n3.nabble.com/Replicating-data-from-one-ES-cluster-to-another-td4034172.html](http://elasticsearch-users.115913.n3.nabble.com/Replicating-data-from-one-ES-cluster-to-another-td4034172.html)) > use rsync ([http://rsync.samba.org/](http://rsync.samba.org/)) to update the underlying Lucene index > files between clusters once an update is complete, so we could have > development, test and live clusters and use rsync to push data between them > as required. I've worked at a relatively low level with Lucene before and > am broadly happy that this approach would work, but can anyone provide any > experience of trying it? > > I've searched for an elasticsearch-to-elasticsearch river but have not > been able to find one. If one exists that would obviously be a good option. > Does anyone know of one? > > Broadly, I can't beleive that our usage pattern of separated dev -\> test > -\> live with on demand updates has not been used elsewhere, and that this > problem hasn't been solved before. Does anyone have any experience of > solving this? Have we missed something obvious here? > > Thanks > > Alex > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:17am UTC](https://discuss.elastic.co/t/deploy-develpment-test-and-live-instances/13553/4 "2017-07-06T02:17:06Z")