Upgrade woes: JDK not accessible, snapshots slow

Elastic Stack Elasticsearch
1

I'm trying to upgrade our production cluster from 5.6.2 to 6.x and it's not going well.

Firstly, snapshotting to S3 was taking forever (~51h for 1TB). After spending some time talking to AWS support with no results I remembered reading something about the latest 5.x versions having better throughput for S3 snapshotting, so I took down a node to start upgrading it.

I had a little trouble upgrading the JDK, so uninstalled it and tried to reinstall. It turns out that not only have Oracle permanently moved / broken links to older versions of the JDK, currently (24th March) even download links to the most recent JDK versions lead to 404s on Oracle's site: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

So now that I have a node down I can't even snapshot at the slower rate, my upgrade window is screwed :confused: and I've no idea when Oracle will get their :poop: together

No response required, just needed to vent!

2

In an hilarious twist, before uninstalling Java I bounced the server to allow some pending security updates to be applied.

Of course Elasticsearch automatically started when this happened, so although I've removed java from the machine, the process / node is still happily running, as is the snapshot, because it was never cancelled.

So if I want to have this available for Monday, I think possibly the steps I have to take are:

  1. Cancel the snapshot
  2. Allow the java-less node to die and carry on with the two remaining data nodes (configuration is one master (no data) and three data (no master). Not sure if this is possible, if one node needs to be down while upgrading?
  3. Create a drive on EC2 and create the snapshot there (not sure if another plugin is needed for this)

Any thoughts welcome :slight_smile:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.