Advices on migrating 1.3.2 to 1.4.1

Hello

We currently have a cluster with 50 millions of docs using ElasticSearch
version 1.3.2

We were looking for something like a persisted filter, and the filtered
aliases, added in version 1.4.0.beta1, seems perfect for it

Our infrastructure team is not happy to upgrading it in production without
doing a lot of tests before, so we have to do a lot of tests and upgrade
later

We are looking for some advices in what can go wrong with this upgrade,
what are the risks?

And also, is there a way to implement a "persistent filter" in our current
version? I mean, some of our users will have access to a part of our data,
we need something like a database view. We could send a filter in every
request, but that would be too slow with, let's say, 10 millions of ids.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJp2533WXXhA4hAd4qBPWa0ZUZGBPUFQ0V4Tv_u7p2OuyCChoA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

1.4 changed a lot of things, especially at the distributed system level, so
testing it in your staging environment will certainly help ensure that
things work as expected.

Filtered aliases have been available for a long time (even before
1.4.0.beta1), it's very likely that they are already available with the
current version that you are running. However, a filter containing 10
million of ids will be slow anyway, even if you cache them the first
execution on a new segment might cause latency spikes since there are lots
of postings lists that need to be merged. Would it be possible to change it
to a simpler term filter, eg. by adding more metadata to your documents?

On Mon, Dec 1, 2014 at 9:23 PM, Roger de Cordova Farias <
roger.farias@fontec.inf.br> wrote:

Hello

We currently have a cluster with 50 millions of docs using ElasticSearch
version 1.3.2

We were looking for something like a persisted filter, and the filtered
aliases, added in version 1.4.0.beta1, seems perfect for it

Our infrastructure team is not happy to upgrading it in production without
doing a lot of tests before, so we have to do a lot of tests and upgrade
later

We are looking for some advices in what can go wrong with this upgrade,
what are the risks?

And also, is there a way to implement a "persistent filter" in our current
version? I mean, some of our users will have access to a part of our data,
we need something like a database view. We could send a filter in every
request, but that would be too slow with, let's say, 10 millions of ids.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJp2533WXXhA4hAd4qBPWa0ZUZGBPUFQ0V4Tv_u7p2OuyCChoA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJp2533WXXhA4hAd4qBPWa0ZUZGBPUFQ0V4Tv_u7p2OuyCChoA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6LV9v5uzeaiKwTXuZMhihhTrhKwmS0cPVmsLfGfLKYjw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thank you for your response

Looks like I read it wrong in the documentation, only the "Fields referred
to in alias filters must exist in the mappings of the index/indices pointed
to by the alias." part was included in the 1.4.0.beta1

Anyway, I found the terms lookup mechanism
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html#_terms_lookup_mechanism
that
also solves our problem of sending a big filter in every request.

What we are doing is that the user, after doing a search, can "bookmark"
the results. Then he has the possibility to do new searchs on his
bookmarked docs only.

Adding metadata to our docs with information of who bookmarked them would
work, too... it only will be harder to update, because the user can
bookmark/un-bookmark them on the flow and in batches (like "bookmark all
docs of the search result")

I will study the approaches to see wich one fits better for us

Thank you very much

2014-12-03 11:46 GMT-02:00 Adrien Grand adrien.grand@elasticsearch.com:

Hi,

1.4 changed a lot of things, especially at the distributed system level,
so testing it in your staging environment will certainly help ensure that
things work as expected.

Filtered aliases have been available for a long time (even before
1.4.0.beta1), it's very likely that they are already available with the
current version that you are running. However, a filter containing 10
million of ids will be slow anyway, even if you cache them the first
execution on a new segment might cause latency spikes since there are lots
of postings lists that need to be merged. Would it be possible to change it
to a simpler term filter, eg. by adding more metadata to your documents?

On Mon, Dec 1, 2014 at 9:23 PM, Roger de Cordova Farias <
roger.farias@fontec.inf.br> wrote:

Hello

We currently have a cluster with 50 millions of docs using ElasticSearch
version 1.3.2

We were looking for something like a persisted filter, and the filtered
aliases, added in version 1.4.0.beta1, seems perfect for it

Our infrastructure team is not happy to upgrading it in production
without doing a lot of tests before, so we have to do a lot of tests and
upgrade later

We are looking for some advices in what can go wrong with this upgrade,
what are the risks?

And also, is there a way to implement a "persistent filter" in our
current version? I mean, some of our users will have access to a part of
our data, we need something like a database view. We could send a filter in
every request, but that would be too slow with, let's say, 10 millions of
ids.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJp2533WXXhA4hAd4qBPWa0ZUZGBPUFQ0V4Tv_u7p2OuyCChoA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJp2533WXXhA4hAd4qBPWa0ZUZGBPUFQ0V4Tv_u7p2OuyCChoA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6LV9v5uzeaiKwTXuZMhihhTrhKwmS0cPVmsLfGfLKYjw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6LV9v5uzeaiKwTXuZMhihhTrhKwmS0cPVmsLfGfLKYjw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJp2533GRyUM4vwWcLKkOAuGLQqc13%3D1w8T%2BDHZWFEN7CGs-Gg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I upgraded our logging cluster to 1.4 without any problems.

When I looked into upgrading a separate dev/test instance used for a
different purpose I ran into problems with the plugins. If you are using
plugins, make sure they are supported in 1.4.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1873d1cb-6f49-413d-8157-1220b64411e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thank you for the advice

2014-12-04 9:30 GMT-02:00 Elvar Böðvarsson elvarb@gmail.com:

I upgraded our logging cluster to 1.4 without any problems.

When I looked into upgrading a separate dev/test instance used for a
different purpose I ran into problems with the plugins. If you are using
plugins, make sure they are supported in 1.4.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1873d1cb-6f49-413d-8157-1220b64411e0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1873d1cb-6f49-413d-8157-1220b64411e0%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJp2533%3D%3Dr1d__%2BKgQr%2Ba66rQ%3Df4WEgMNwphAY0hu06APomqeA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.