Lucene 4.0

Just wondering if any progress has been made on not requiring full cluster
restarts. Version 1.0? :slight_smile:

--
Ivan

On Thu, Oct 11, 2012 at 6:16 PM, Shay Banon kimchy@gmail.com wrote:

Heya, few points:

Indexes in elasticsearch has been "backward" compatible, and when
upgrading to Lucene 4.0, it will be backward compatible as well (both on
the Lucene level, and on the ES level). You might need to do an "upgrade"
to make use of newer versions, or to make sure there is no "emulation"
layer on Lucene level.

So, upgrading to 0.20 and 0.21 will be compatible index wise (no need to
reindex), but you will need to do a full cluster restart (though we are
working on eventually not needing that as well, first infrastructure for
that is already going to be in upcoming 0.20).

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Between major versions, we hope to nail it for 1.0, but the plan is not to hold 1.0 is we won't have done by that time. We'll see...

On Mar 1, 2013, at 9:48 AM, Ivan Brusic ivan@brusic.com wrote:

Just wondering if any progress has been made on not requiring full cluster restarts. Version 1.0? :slight_smile:

--
Ivan

On Thu, Oct 11, 2012 at 6:16 PM, Shay Banon kimchy@gmail.com wrote:
Heya, few points:

Indexes in elasticsearch has been "backward" compatible, and when upgrading to Lucene 4.0, it will be backward compatible as well (both on the Lucene level, and on the ES level). You might need to do an "upgrade" to make use of newer versions, or to make sure there is no "emulation" layer on Lucene level.

So, upgrading to 0.20 and 0.21 will be compatible index wise (no need to reindex), but you will need to do a full cluster restart (though we are working on eventually not needing that as well, first infrastructure for that is already going to be in upcoming 0.20).

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for following up on this thread. The obvious next question is will
1.0 be the successor to 0.90?

--
Ivan

On Mon, Mar 4, 2013 at 4:46 PM, kimchy@gmail.com wrote:

Between major versions, we hope to nail it for 1.0, but the plan is not to
hold 1.0 is we won't have done by that time. We'll see...

On Mar 1, 2013, at 9:48 AM, Ivan Brusic ivan@brusic.com wrote:

Just wondering if any progress has been made on not requiring full cluster
restarts. Version 1.0? :slight_smile:

--
Ivan

On Thu, Oct 11, 2012 at 6:16 PM, Shay Banon kimchy@gmail.com wrote:

Heya, few points:

Indexes in elasticsearch has been "backward" compatible, and when
upgrading to Lucene 4.0, it will be backward compatible as well (both on
the Lucene level, and on the ES level). You might need to do an "upgrade"
to make use of newer versions, or to make sure there is no "emulation"
layer on Lucene level.

So, upgrading to 0.20 and 0.21 will be compatible index wise (no need to
reindex), but you will need to do a full cluster restart (though we are
working on eventually not needing that as well, first infrastructure for
that is already going to be in upcoming 0.20).

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Yes, 1.0 will be the next major release post 0.90. Will post our thoughts on which features we wish to try and get into 1.0 once we release 0.90.

On Mar 4, 2013, at 5:12 PM, Ivan Brusic ivan@brusic.com wrote:

Thanks for following up on this thread. The obvious next question is will 1.0 be the successor to 0.90?

--
Ivan

On Mon, Mar 4, 2013 at 4:46 PM, kimchy@gmail.com wrote:
Between major versions, we hope to nail it for 1.0, but the plan is not to hold 1.0 is we won't have done by that time. We'll see...

On Mar 1, 2013, at 9:48 AM, Ivan Brusic ivan@brusic.com wrote:

Just wondering if any progress has been made on not requiring full cluster restarts. Version 1.0? :slight_smile:

--
Ivan

On Thu, Oct 11, 2012 at 6:16 PM, Shay Banon kimchy@gmail.com wrote:
Heya, few points:

Indexes in elasticsearch has been "backward" compatible, and when upgrading to Lucene 4.0, it will be backward compatible as well (both on the Lucene level, and on the ES level). You might need to do an "upgrade" to make use of newer versions, or to make sure there is no "emulation" layer on Lucene level.

So, upgrading to 0.20 and 0.21 will be compatible index wise (no need to reindex), but you will need to do a full cluster restart (though we are working on eventually not needing that as well, first infrastructure for that is already going to be in upcoming 0.20).

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Can you add an option that disables this at the cost of making your indices
not backwards compatible? (but faster/ memory friendlier)
Is there a way of completely upgrading your index planned?

On Friday, October 12, 2012 12:26:33 PM UTC+4, simonw wrote:

On Thursday, October 11, 2012 10:40:53 PM UTC+2, Ivan Brusic wrote:

Lucene is always backwards compatible within one version, so Lucene 4.0
code should be able to read a 3.x index. Writing indexes with a newer
version and reading them in an older one is more problematic. That said, a
new ES version is more than just a Lucene upgrade, so it might require a
reindex.

this question has been asked a couple of times so let me elaborate on this
a little for those who are interested what the main differences are between
4.0 and 3.x on the index level. It is correct in general that lucene 4.0
can read 3.0 indexes but this comes with a cost this time. in lucene 4.0 we
changed the sort order from UTF-16 to UTF-8 on the lowest level so 3.x
indices are sorted "differently". To make this still work with the 4.0 API
we added a "re-mapping" layer for surrogate characters to maintain the
correct sort order. This will have some cost in performance but it should
not be dramatic. Yet, it is still a good idea to upgrade the index. The
good news is lucene can by-itself do that in the background. If you merge a
3.x segment with 4.0 it will be merged into a 4.0 segment so you can
"upgrade-over-time". Anyhow, in practice ES users should be need this
knowledge and we will provide flexible ways to upgrade to an ES version
that runs lucene 4.0.

simon

ES version incompatibility is always due to the internal communication
API changing.

--
Ivan

On Thu, Oct 11, 2012 at 11:20 AM, Eugene Strokin eug...@strokin.infowrote:

Shay, thanks for update.
Just wander about back compatibility.
If switching from 0.19 to 0.20 will it require reindexing, or we could
just update the libs?
And from 20 to 21, I guess it will require reindexing, but want to hear
confirmation from you.

Thank you

On Thursday, October 11, 2012 11:18:46 AM UTC-4, kimchy wrote:

Heya fellows,

Lucene 4.0 is out the door, wanted to update you all about how we 

plan to upgrade to it on elasticsearch. The plan is to first release 0.20
version still using Lucene 3.6.x, this should happen in the next week or
so. The reason is that we want to get 0.20 features at the hand of users as
fast as possible, without waiting for the upgrade to 4.0.

We will also start the process of upgrading the 4.0, which will be 

in the next major elasticsearch version. This will include upgrading to
4.0, and making use of the new features and exposing them to the users.
This shouldn't take too long, though we do want to see 4.0.0 GA Lucene "out
in the wild" a bit before releasing a formal release (0.21) of
elasticsearch with it.

-shay.banon

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Tuesday, March 5, 2013 11:14:18 AM UTC+1, Mo wrote:

Can you add an option that disables this at the cost of making your
indices not backwards compatible? (but faster/ memory friendlier)
Is there a way of completely upgrading your index planned?

I am not sure what you mean by "disables this"? If you want to upgrade your
entire index to Lucene 4.0 you need to run an optimze on the index which
will re-write the index in the new format.

simon

On Friday, October 12, 2012 12:26:33 PM UTC+4, simonw wrote:

On Thursday, October 11, 2012 10:40:53 PM UTC+2, Ivan Brusic wrote:

Lucene is always backwards compatible within one version, so Lucene 4.0
code should be able to read a 3.x index. Writing indexes with a newer
version and reading them in an older one is more problematic. That said, a
new ES version is more than just a Lucene upgrade, so it might require a
reindex.

this question has been asked a couple of times so let me elaborate on
this a little for those who are interested what the main differences are
between 4.0 and 3.x on the index level. It is correct in general that
lucene 4.0 can read 3.0 indexes but this comes with a cost this time. in
lucene 4.0 we changed the sort order from UTF-16 to UTF-8 on the lowest
level so 3.x indices are sorted "differently". To make this still work with
the 4.0 API we added a "re-mapping" layer for surrogate characters to
maintain the correct sort order. This will have some cost in performance
but it should not be dramatic. Yet, it is still a good idea to upgrade the
index. The good news is lucene can by-itself do that in the background. If
you merge a 3.x segment with 4.0 it will be merged into a 4.0 segment so
you can "upgrade-over-time". Anyhow, in practice ES users should be need
this knowledge and we will provide flexible ways to upgrade to an ES
version that runs lucene 4.0.

simon

ES version incompatibility is always due to the internal communication
API changing.

--
Ivan

On Thu, Oct 11, 2012 at 11:20 AM, Eugene Strokin eug...@strokin.infowrote:

Shay, thanks for update.
Just wander about back compatibility.
If switching from 0.19 to 0.20 will it require reindexing, or we could
just update the libs?
And from 20 to 21, I guess it will require reindexing, but want to hear
confirmation from you.

Thank you

On Thursday, October 11, 2012 11:18:46 AM UTC-4, kimchy wrote:

Heya fellows,

Lucene 4.0 is out the door, wanted to update you all about how we 

plan to upgrade to it on elasticsearch. The plan is to first release 0.20
version still using Lucene 3.6.x, this should happen in the next week or
so. The reason is that we want to get 0.20 features at the hand of users as
fast as possible, without waiting for the upgrade to 4.0.

We will also start the process of upgrading the 4.0, which will be 

in the next major elasticsearch version. This will include upgrading to
4.0, and making use of the new features and exposing them to the users.
This shouldn't take too long, though we do want to see 4.0.0 GA Lucene "out
in the wild" a bit before releasing a formal release (0.21) of
elasticsearch with it.

-shay.banon

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.