Upgrade 0.26.6 -> 1.2.2 any catches?


(Scott Decker) #1

Hey all,
We are going to start our battle plans for upgrading from 0.26 to 1.2.2
and I have been reading through this doc:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/breaking-changes.html

to help see what things we may need to worry about.

Some specific questions
1 - Can the logging.yml file now be done as json like the elasticsearch
one? Sorry, kind of hate the yml format, and json is much easier. here is
hoping....
2 - I know our index mappings need to be updated/checked, but anything pop
out as to really look for? Like, on standard analyzer something was set
before, but now it is gone kind of things?
3 - startup. we have just been running ./bin/elasticsearch. I see that
this has now changed. any recommended ways to have similar function as that
startup?
4 - index formats? Pretty sure that when we upgrade, the indexes are redone
in a new format, yes? Or, if we had static indexes, and we upgrade, are
some in an older format and new ones in the new format?
4a - if they are being re-written, is the limit more IO than anything else?
So, if we had say 1 TB of indexes to do, we should expect the process to
take while to update/write the new index files
5 - any other gotchas people have experienced? Recommend to just go from
.26 -> 1.2.2? Just making sure!

We are looking forward to finally being able to get to this so we can also
check out Marvel on real work being done. Should be fun and thanks for any
insights everyone has!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/66ff929d-1f36-406b-a53a-e82545abc98a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

You probably want to go from 0.20.X to 0.90.12, then to 1.X to be safe.
Ensure you do a optimise (merge) on each index so that they are updated to
the newer lucene version number, then when they are all done move to the
next major version.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 15 July 2014 08:20, Scott Decker scott@publishthis.com wrote:

Hey all,
We are going to start our battle plans for upgrading from 0.26 to 1.2.2
and I have been reading through this doc:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/breaking-changes.html

to help see what things we may need to worry about.

Some specific questions
1 - Can the logging.yml file now be done as json like the elasticsearch
one? Sorry, kind of hate the yml format, and json is much easier. here is
hoping....
2 - I know our index mappings need to be updated/checked, but anything pop
out as to really look for? Like, on standard analyzer something was set
before, but now it is gone kind of things?
3 - startup. we have just been running ./bin/elasticsearch. I see that
this has now changed. any recommended ways to have similar function as that
startup?
4 - index formats? Pretty sure that when we upgrade, the indexes are
redone in a new format, yes? Or, if we had static indexes, and we upgrade,
are some in an older format and new ones in the new format?
4a - if they are being re-written, is the limit more IO than anything
else? So, if we had say 1 TB of indexes to do, we should expect the process
to take while to update/write the new index files
5 - any other gotchas people have experienced? Recommend to just go from
.26 -> 1.2.2? Just making sure!

We are looking forward to finally being able to get to this so we can also
check out Marvel on real work being done. Should be fun and thanks for any
insights everyone has!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/66ff929d-1f36-406b-a53a-e82545abc98a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/66ff929d-1f36-406b-a53a-e82545abc98a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YPvT%3DPu5haH2EhwOzWnEbW%2B6s8kxOEO-zfqMTwiS5UVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #3

First of all, there is no version 0.26. I am assuming you meant 0.20.6.
Either way, any upgrade from prior of 1.0 to 1.x will require a full
cluster restart.

  1. No clue

  2. Many settings like omit_norms were deprecated, but are still support. I
    think that omit_tf has been changed.

  3. I would install via apt-get/yum or use the elasticearch wrapper

  4. Lucene can read indicies up to 1 version behind, so since your index is
    based on Lucene 3 and Elasticsearch 1.x is Lucene, you do not need to
    upgrade your indices. You should run an optimize on your indices when the
    cluster starts back up so that Lucene can upgrade the existing segments.

4a - You can skip the optimize, but you really shouldn't. :slight_smile:

  1. One gotcha that I can think about is that all stores are now throttled
    with a very low level (IMHO). Increase it based on your hardware and
    indexing requirements. The syntax for disabling allocations has change. I'm
    sure there are other gotchas, but most settings are still the same.

Cheers,

Ivan

On Mon, Jul 14, 2014 at 3:20 PM, Scott Decker scott@publishthis.com wrote:

Hey all,
We are going to start our battle plans for upgrading from 0.26 to 1.2.2
and I have been reading through this doc:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/breaking-changes.html

to help see what things we may need to worry about.

Some specific questions
1 - Can the logging.yml file now be done as json like the elasticsearch
one? Sorry, kind of hate the yml format, and json is much easier. here is
hoping....
2 - I know our index mappings need to be updated/checked, but anything pop
out as to really look for? Like, on standard analyzer something was set
before, but now it is gone kind of things?
3 - startup. we have just been running ./bin/elasticsearch. I see that
this has now changed. any recommended ways to have similar function as that
startup?
4 - index formats? Pretty sure that when we upgrade, the indexes are
redone in a new format, yes? Or, if we had static indexes, and we upgrade,
are some in an older format and new ones in the new format?
4a - if they are being re-written, is the limit more IO than anything
else? So, if we had say 1 TB of indexes to do, we should expect the process
to take while to update/write the new index files
5 - any other gotchas people have experienced? Recommend to just go from
.26 -> 1.2.2? Just making sure!

We are looking forward to finally being able to get to this so we can also
check out Marvel on real work being done. Should be fun and thanks for any
insights everyone has!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/66ff929d-1f36-406b-a53a-e82545abc98a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/66ff929d-1f36-406b-a53a-e82545abc98a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAoYBqBsx%3DDLe_cQy66YXY_UxxrVUNVsJD%2BXr9KMEu7zQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(smonasco-2) #4

@Ivan, what do you mean the stores are throttled with a very low level? Do you mean index threads or merge factor or what?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/63273182-325a-4253-bb79-f9a7b5e3b199%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Scott Decker) #5

Very helpful everyone! (and yes, should have been .20.6!

so, bigger takeaways are
1 - see if we move from 20.6 to .90 and then to 1.2.2 (will add this to our
conversations)
2 - optimize! that is a good one to note. had forgotten the internals of
lucene while talking all of our steps. thanks for reminding me!

Can we run with the same elasticsearch.yml/json file? has anything changed
in the config part that I should be aware of? I know sometimes the
name.value pair naming changes, but, hard to find out what those are
without code digging or trying it and seeing what fails.

On Monday, July 14, 2014 3:37:01 PM UTC-7, Ivan Brusic wrote:

First of all, there is no version 0.26. I am assuming you meant 0.20.6.
Either way, any upgrade from prior of 1.0 to 1.x will require a full
cluster restart.

  1. No clue

  2. Many settings like omit_norms were deprecated, but are still support. I
    think that omit_tf has been changed.

  3. I would install via apt-get/yum or use the elasticearch wrapper

  4. Lucene can read indicies up to 1 version behind, so since your index is
    based on Lucene 3 and Elasticsearch 1.x is Lucene, you do not need to
    upgrade your indices. You should run an optimize on your indices when the
    cluster starts back up so that Lucene can upgrade the existing segments.

4a - You can skip the optimize, but you really shouldn't. :slight_smile:

  1. One gotcha that I can think about is that all stores are now throttled
    with a very low level (IMHO). Increase it based on your hardware and
    indexing requirements. The syntax for disabling allocations has change. I'm
    sure there are other gotchas, but most settings are still the same.

Cheers,

Ivan

On Mon, Jul 14, 2014 at 3:20 PM, Scott Decker <sc...@publishthis.com
<javascript:>> wrote:

Hey all,
We are going to start our battle plans for upgrading from 0.26 to 1.2.2
and I have been reading through this doc:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/breaking-changes.html

to help see what things we may need to worry about.

Some specific questions
1 - Can the logging.yml file now be done as json like the elasticsearch
one? Sorry, kind of hate the yml format, and json is much easier. here is
hoping....
2 - I know our index mappings need to be updated/checked, but anything
pop out as to really look for? Like, on standard analyzer something was set
before, but now it is gone kind of things?
3 - startup. we have just been running ./bin/elasticsearch. I see that
this has now changed. any recommended ways to have similar function as that
startup?
4 - index formats? Pretty sure that when we upgrade, the indexes are
redone in a new format, yes? Or, if we had static indexes, and we upgrade,
are some in an older format and new ones in the new format?
4a - if they are being re-written, is the limit more IO than anything
else? So, if we had say 1 TB of indexes to do, we should expect the process
to take while to update/write the new index files
5 - any other gotchas people have experienced? Recommend to just go from
.26 -> 1.2.2? Just making sure!

We are looking forward to finally being able to get to this so we can
also check out Marvel on real work being done. Should be fun and thanks for
any insights everyone has!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/66ff929d-1f36-406b-a53a-e82545abc98a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/66ff929d-1f36-406b-a53a-e82545abc98a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/26e360db-953a-48e6-acec-2ecd1eae2ede%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #6

Read more about it here:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html#store-throttling

Previously it was unbounded, but now the default is 20mb, which I found to
be extremely low. Also, prior to 1.2, there was a Lucene bug which made
throttling even more limiting:

As far as settings goes, I can't think of any that have changed. Perhaps if
you posted your altered settings, others can give it a once over.

BTW, there is no need for the @ symbol when writing on a mailing list.
Normal English writing rules apply, not the Twitter ones.

Cheers,

Ivan

On Mon, Jul 14, 2014 at 8:41 PM, smonasco smonasco@gmail.com wrote:

@Ivan, what do you mean the stores are throttled with a very low level?
Do you mean index threads or merge factor or what?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/63273182-325a-4253-bb79-f9a7b5e3b199%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAc9uaBvsH56y31KnaPHGJ67GMXsuTOsBwF7hn7N7PrCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #7