# Upgrading Elasticsearch from 19.8 -\>20.1, S3 Gateway -\> Local **URL:** \ **Category:** Elasticsearch\ **Created:** [December 10, 2012, 7:51pm UTC](https://discuss.elastic.co/t/upgrading-elasticsearch-from-19-8-20-1-s3-gateway-local/10021 "2012-12-10T19:51:18Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Neil\_Moonka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neil_moonka/32/2584_2.png) [@Neil\_Moonka](https://discuss.elastic.co/u/Neil_Moonka)\ **Post date:** [December 10, 2012, 7:51pm UTC](https://discuss.elastic.co/t/upgrading-elasticsearch-from-19-8-20-1-s3-gateway-local/10021/1 "2012-12-10T19:51:18Z")
We are looking to upgrade our ES instances to 20.1 from 19.8. We have about 7 aws nodes in a cluster, with about 25 Million docs indexed. I see that the Shared gateway has been deprecated in favor of the local, is there a way to migrate over the gateway? From what I can tell, the only alternative is to boot up another cluster which uses a local gateway and copy over each of the documents, is there an easier way to do so? -- ---
**Author:** ![karmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karmi/32/44951_2.png) [@karmi](https://discuss.elastic.co/u/karmi)\ **Post date:** [December 11, 2012, 9:19am UTC](https://discuss.elastic.co/t/upgrading-elasticsearch-from-19-8-20-1-s3-gateway-local/10021/2 "2012-12-11T09:19:46Z")
You can migrate the data using Elasticsearch itself, some guidelines from earlier reply [[Redirecting to Google Groups](https://groups.google.com/forum/#!msg/elasticsearch/sWp9XDzNmk8/YFRVSFtWFE4J)]: - Create a two new IOPS EBS volumes [1], with enough space to hold your data - Launch a new EC2 instance with proper security groups - Mount the EBS on the new instance [2], to a good location such as `/usr/local/var/elasticsearch/data1` - Install and configure elasticsearch on the machine, using the same cluster name as your original cluster, using a `local` gateway, pointed to the location where you mounted the EBS volume - Launch elasticsearch on these new instances - Increase the `number_of_replicas` for your indices to four (ie. equal to the number of nodes). Your data will be now spread across all the nodes: the old ones, and the new ones. - Use Paramedic, BigDesk or Head elasticsearch plugins to monitor cluster health: once you're in a "green" health, and all shards are allocated, you can shutdown the old, S3-based nodes - You have migrated all data to a new cluster. The best practice now would be to do a snapshot of your EBS volumes, so you have a recovery strategy. You can delete the S3 buckets after doing that. This strategy allows you to scale when the volume of your data grows and the computing capacity of your cluster is enough: you can create a new set of EBS volumes, mount them to a location such as `/usr/local/var/elasticsearch/data2` and point elasticsearch `data.path` to both locations (it is possible to use multiple directories as the `data.path`). I'm currently working on having EBS support like that in the Chef cookbook [[GitHub - karmi/cookbook-elasticsearch: https://github.com/elasticsearch/cookbook-elasticsearch](https://github.com/karmi/cookbook-elasticsearch/)], so it should me easier to quickly whip up clusters on AWS in the near future. Karel On Monday, December 10, 2012 8:51:18 PM UTC+1, Neil Moonka wrote: > We are looking to upgrade our ES instances to 20.1 from 19.8. We have > about 7 aws nodes in a cluster, with about 25 Million docs indexed. I see > that the Shared gateway has been deprecated in favor of the local, is there > a way to migrate over the gateway? From what I can tell, the only > alternative is to boot up another cluster which uses a local gateway and > copy over each of the documents, is there an easier way to do so? -- ---
**Author:** ![Neil\_Moonka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neil_moonka/32/2584_2.png) [@Neil\_Moonka](https://discuss.elastic.co/u/Neil_Moonka)\ **Post date:** [December 12, 2012, 1:14am UTC](https://discuss.elastic.co/t/upgrading-elasticsearch-from-19-8-20-1-s3-gateway-local/10021/3 "2012-12-12T01:14:17Z")
Thanks! This essentially worked, though I did a modification of it. We had added a couple of extra nodes recently to support some upcoming activities, so we are operating with a little breathing room. We turned off a box, pointed it to local, launched it, and then let it replicate (upping the number\_of\_replicas). We then worked our way through the cluster bouncing them and pointing to local in a rolling restart fashion. This let us avoid any downtime, thanks! -Neil On Tuesday, December 11, 2012 1:19:46 AM UTC-8, Karel Minařík wrote: > You can migrate the data using Elasticsearch itself, some guidelines from > earlier reply [ > [Redirecting to Google Groups](https://groups.google.com/forum/#!msg/elasticsearch/sWp9XDzNmk8/YFRVSFtWFE4J) > ]: > > - Create a two new IOPS EBS volumes [1], with enough space to hold your > data > - Launch a new EC2 instance with proper security groups > - Mount the EBS on the new instance [2], to a good location such as > `/usr/local/var/elasticsearch/data1` > - Install and configure elasticsearch on the machine, using the same > cluster name as your original cluster, using a `local` gateway, pointed to > the location where you mounted the EBS volume > - Launch elasticsearch on these new instances > - Increase the `number_of_replicas` for your indices to four (ie. equal to > the number of nodes). Your data will be now spread across all the nodes: > the old ones, and the new ones. > - Use Paramedic, BigDesk or Head elasticsearch plugins to monitor cluster > health: once you're in a "green" health, and all shards are allocated, you > can shutdown the old, S3-based nodes > - You have migrated all data to a new cluster. The best practice now would > be to do a snapshot of your EBS volumes, so you have a recovery strategy. > You can delete the S3 buckets after doing that. > > This strategy allows you to scale when the volume of your data grows and > the computing capacity of your cluster is enough: you can create a new set > of EBS volumes, mount them to a location such as > `/usr/local/var/elasticsearch/data2` and point elasticsearch `data.path` to > both locations (it is possible to use multiple directories as the > `data.path`). > > I'm currently working on having EBS support like that in the Chef cookbook > [[GitHub - karmi/cookbook-elasticsearch: https://github.com/elasticsearch/cookbook-elasticsearch](https://github.com/karmi/cookbook-elasticsearch/)], so it should me > easier to quickly whip up clusters on AWS in the near future. > > Karel > > On Monday, December 10, 2012 8:51:18 PM UTC+1, Neil Moonka wrote: > > > We are looking to upgrade our ES instances to 20.1 from 19.8. We have > > about 7 aws nodes in a cluster, with about 25 Million docs indexed. I see > > that the Shared gateway has been deprecated in favor of the local, is there > > a way to migrate over the gateway? From what I can tell, the only > > alternative is to boot up another cluster which uses a local gateway and > > copy over each of the documents, is there an easier way to do so? -- ---
**Author:** ![karmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karmi/32/44951_2.png) [@karmi](https://discuss.elastic.co/u/karmi)\ **Post date:** [December 12, 2012, 10:03am UTC](https://discuss.elastic.co/t/upgrading-elasticsearch-from-19-8-20-1-s3-gateway-local/10021/4 "2012-12-12T10:03:47Z")
Hello, yes, that sounds like a good approach. Note that maybe you could use the `auto_expand_replicas` setting [[A setting to auto expand the number of replicas of an index (based on data nodes) · Issue #623 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/623), [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html)] to automatically increase replicas when a node is added, and decrease them if node is shut down, minimizing all the administration around the workflow a bit. Karel On Dec 12, 2012, at 2:14 AM, Neil Moonka [nkmoonka@gmail.com](mailto:nkmoonka@gmail.com) wrote: > Thanks! This essentially worked, though I did a modification of it. We had added a couple of extra nodes recently to support some upcoming activities, so we are operating with a little breathing room. We turned off a box, pointed it to local, launched it, and then let it replicate (upping the number\_of\_replicas). We then worked our way through the cluster bouncing them and pointing to local in a rolling restart fashion. This let us avoid any downtime, thanks! > > -Neil > > On Tuesday, December 11, 2012 1:19:46 AM UTC-8, Karel Minařík wrote: > You can migrate the data using Elasticsearch itself, some guidelines from earlier reply [[Redirecting to Google Groups](https://groups.google.com/forum/#!msg/elasticsearch/sWp9XDzNmk8/YFRVSFtWFE4J)]: > > - Create a two new IOPS EBS volumes [1], with enough space to hold your data > - Launch a new EC2 instance with proper security groups > - Mount the EBS on the new instance [2], to a good location such as `/usr/local/var/elasticsearch/data1` > - Install and configure elasticsearch on the machine, using the same cluster name as your original cluster, using a `local` gateway, pointed to the location where you mounted the EBS volume > - Launch elasticsearch on these new instances > - Increase the `number_of_replicas` for your indices to four (ie. equal to the number of nodes). Your data will be now spread across all the nodes: the old ones, and the new ones. > - Use Paramedic, BigDesk or Head elasticsearch plugins to monitor cluster health: once you're in a "green" health, and all shards are allocated, you can shutdown the old, S3-based nodes > - You have migrated all data to a new cluster. The best practice now would be to do a snapshot of your EBS volumes, so you have a recovery strategy. You can delete the S3 buckets after doing that. > > This strategy allows you to scale when the volume of your data grows and the computing capacity of your cluster is enough: you can create a new set of EBS volumes, mount them to a location such as `/usr/local/var/elasticsearch/data2` and point elasticsearch `data.path` to both locations (it is possible to use multiple directories as the `data.path`). > > I'm currently working on having EBS support like that in the Chef cookbook [[GitHub - karmi/cookbook-elasticsearch: https://github.com/elasticsearch/cookbook-elasticsearch](https://github.com/karmi/cookbook-elasticsearch/)], so it should me easier to quickly whip up clusters on AWS in the near future. > > Karel > > On Monday, December 10, 2012 8:51:18 PM UTC+1, Neil Moonka wrote: > We are looking to upgrade our ES instances to 20.1 from 19.8. We have about 7 aws nodes in a cluster, with about 25 Million docs indexed. I see that the Shared gateway has been deprecated in favor of the local, is there a way to migrate over the gateway? From what I can tell, the only alternative is to boot up another cluster which uses a local gateway and copy over each of the documents, is there an easier way to do so? > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:00am UTC](https://discuss.elastic.co/t/upgrading-elasticsearch-from-19-8-20-1-s3-gateway-local/10021/5 "2017-07-06T03:00:10Z")