Upgrading/changing JVM

Hi,

What's the best approach for upgrading the JVM used by a cluster?

We're currently using the OpenJDK as packaged with Ubuntu 12.04 (1.6.0_24, fwiw), but I do keep reading that Oracle's latest Java 7 is the way to go. Obviously I'll test any upgrades on a test cluster first, but it'd be good to know what direction I should head in!

Can I upgrade the JVM on one machine at a time, and basically do a rolling restart of the cluster? Would you expect anything to break if different nodes are using different JVM versions, or using data files created from ES on the older JVM with a newer one?

I'm using ES 0.19.8.

Cheers,
Dan

--
Dan Fairs | dan.fairs@gmail.com | @danfairs | secondsync.com

--

Hi Dan,

it's tricky. I know there are some installations that run with mixed Java,
Version 6 and 7, for a significant time without issues. Note, this can
work, but you should avoid administrative tasks like creating indices,
adding new plugins, and indexing.

A rolling update should work as long as you run only the search API. You
should take care the index is "silent", that is, flushed to the gateway
with no outstanding writes, and no indexing.

I recommended a test: start another cluster instance with a mixed Java 6
and a Java 7 node on a test machine, copy your configuration to them and
start indexing and search. Check if the cluster works.

Due to subtle serialization issues, your ES cluster will not be able to
transport internal Java exceptions in bulk index responses between
different Java 6 and 7 JVM nodes. It is possible you will be affected by
this if you allow the bulk index API while you do the update.

Regarding data files, there are a lot of JVM issues the Lucene committers
are aware of. There had been suggestions a few months before to even block
Lucene from running on blacklisted JVMs. See also
[LUCENE-4276] refuse to execute on broken corrupting jvms - ASF JIRA In summary, there are
some JVMs that are known to harm your index when writing to it, but it is
not easy to identify the version, especially in Ubuntu's Java numbering.
But don't be worried too much, I think the best alternative is to move to
the latest Java 7 JVM. Oracle/OpenJDK are quite aligned in the Java 7
releases (except security fixes unfortunately).

If it helps, when I moved to Java 7, I took the whole cluster down, because
I found it is the fastest method to get the cluster up again.

Cheers,

Jörg

On Monday, December 17, 2012 5:16:49 PM UTC+1, Dan Fairs wrote:

Hi,

What's the best approach for upgrading the JVM used by a cluster?

We're currently using the OpenJDK as packaged with Ubuntu 12.04 (1.6.0_24,
fwiw), but I do keep reading that Oracle's latest Java 7 is the way to go.
Obviously I'll test any upgrades on a test cluster first, but it'd be good
to know what direction I should head in!

Can I upgrade the JVM on one machine at a time, and basically do a rolling
restart of the cluster? Would you expect anything to break if different
nodes are using different JVM versions, or using data files created from ES
on the older JVM with a newer one?

I'm using ES 0.19.8.

Cheers,
Dan

--
Dan Fairs | dan....@gmail.com <javascript:> | @danfairs | secondsync.com

--

Hi Jörg,

Thanks for detailed reply!

A rolling update should work as long as you run only the search API. You should take care the index is "silent", that is, flushed to the gateway with no outstanding writes, and no indexing.

I recommended a test: start another cluster instance with a mixed Java 6 and a Java 7 node on a test machine, copy your configuration to them and start indexing and search. Check if the cluster works.

Right - this was basically my plan.

Due to subtle serialization issues, your ES cluster will not be able to transport internal Java exceptions in bulk index responses between different Java 6 and 7 JVM nodes. It is possible you will be affected by this if you allow the bulk index API while you do the update.

Mm, almost all our index operations are bulk actually; we add up to a million documents per day.

Regarding data files, there are a lot of JVM issues the Lucene committers are aware of. There had been suggestions a few months before to even block Lucene from running on blacklisted JVMs. See also [LUCENE-4276] refuse to execute on broken corrupting jvms - ASF JIRA In summary, there are some JVMs that are known to harm your index when writing to it, but it is not easy to identify the version, especially in Ubuntu's Java numbering. But don't be worried too much, I think the best alternative is to move to the latest Java 7 JVM. Oracle/OpenJDK are quite aligned in the Java 7 releases (except security fixes unfortunately).

Right. This was also my plan! I'm not really a Java person, so understanding the differences/similarities between the Oracle/OpenJDK releases is new to me.

If it helps, when I moved to Java 7, I took the whole cluster down, because I found it is the fastest method to get the cluster up again.

I think that's what I'll do, then. It does seem the safest all round. And, of course, I'll be testing on our staging cluster.

Thanks again.

Cheers,
Dan

Dan Fairs | dan.fairs@gmail.com | @danfairs | secondsync.com

--