How to migrate index without disturbing client?


(Mehmet Özer) #1

Hello,

I am running a cluster which contains many indexes. In the near future I
will have new machines so that I will need to migrate my indexes to those
machines. I have thought some scenarios but it will not be possible to do
so. Let me explain what I thought and why it was not possible;

  • I wanted to put the new machines into existing cluster and then I thought
    I could use exclude._ip option to exclude existing machines. So that old
    indexes would have migrated.

This seemed possible for me but I can not stop working on the index, users
should continue making searches on the indexes. So can you please tell me
how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Nik Everett) #2

What you just described should work fine. exclude._ip will move the shards
off of the nodes you exclude but queries and updates can proceed while this
is happening because the data is still on the old nodes. The updates will
make their way to the new copies via a transaction log reply mechanism.

Depending on how the clients connect you might need to shift their
connection uris to the new servers and that might require restarting them -
depending on how your clients work.

On Fri, May 29, 2015 at 8:38 AM, mehmet özer mehmetozer1988@gmail.com
wrote:

Hello,

I am running a cluster which contains many indexes. In the near future I
will have new machines so that I will need to migrate my indexes to those
machines. I have thought some scenarios but it will not be possible to do
so. Let me explain what I thought and why it was not possible;

  • I wanted to put the new machines into existing cluster and then I
    thought I could use exclude._ip option to exclude existing machines. So
    that old indexes would have migrated.

This seemed possible for me but I can not stop working on the index, users
should continue making searches on the indexes. So can you please tell me
how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1mYXppwguwnPCfyMCdG4jx0AmQBCMoqcxDE%2Bb9U1_nmQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mehmet Özer) #3

Thanks for your answer. I didnt know the exclude doesnt disable the
research. But according to the restart,

  • I cannot restart the cluster because I have 9 tera of data so it can take
    one day to restart all cluster. In addition to that old machines and new
    machines will not know each other( I mean unicast variable of the
    elasticsearch.yml, the old machines will know just old ones). Do you think
    it can cause a problem ?

Thanks

29 Mayıs 2015 Cuma 14:45:40 UTC+2 tarihinde Nikolas Everett yazdı:

What you just described should work fine. exclude._ip will move the shards
off of the nodes you exclude but queries and updates can proceed while this
is happening because the data is still on the old nodes. The updates will
make their way to the new copies via a transaction log reply mechanism.

Depending on how the clients connect you might need to shift their
connection uris to the new servers and that might require restarting them -
depending on how your clients work.

On Fri, May 29, 2015 at 8:38 AM, mehmet özer <mehmeto...@gmail.com
<javascript:>> wrote:

Hello,

I am running a cluster which contains many indexes. In the near future I
will have new machines so that I will need to migrate my indexes to those
machines. I have thought some scenarios but it will not be possible to do
so. Let me explain what I thought and why it was not possible;

  • I wanted to put the new machines into existing cluster and then I
    thought I could use exclude._ip option to exclude existing machines. So
    that old indexes would have migrated.

This seemed possible for me but I can not stop working on the index,
users should continue making searches on the indexes. So can you please
tell me how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Nik Everett) #4

You certainly will have to think of a way to add new cluster masters - but
you can still do it with unicast discovery I think.

Yeah - exclude doesn't disable searches - it just causes elasticsearch to
move the shards using it normal shard moving mechanism. Its really just
adding a routing rule.

You shouldn't have to restart the whole cluster - but you will have to
convince your clients to look at the new nodes.

The last time I did a full restart of the Elasticsearch cluster was about a
year ago when I upgraded from 0.90.something to 1.1.something and I expect
the next full restart to be the upgrade to 2.0 when its released. In the
mean time we've:

  1. Done about twenty rolling restarts to pick up new code
  2. Replaced all the hard drives in all the nodes - two at a time - moving
    data off of the nodes we were restarting using excluse._ip
  3. Changed mount options on the elasticsearch data partition using the same
    technique
  4. More than doubled the number of nodes in the cluster.

In our case the application connects using linux virtual server so all we
had to do is teach the application to retry on connection errors for when
lvs round-robin-ed it to a down Elasticsearch node and teach our lvs pool
maintainer to depool elasticsearch nodes that are down.

I'm not an expert on the Java client so I don't know what you'd have to do
to update and java applications to the new nodes. If you are using a proxy
like nginx you'd just have to change where it points.

Nik

On Fri, May 29, 2015 at 8:51 AM, mehmet özer mehmetozer1988@gmail.com
wrote:

Thanks for your answer. I didnt know the exclude doesnt disable the
research. But according to the restart,

  • I cannot restart the cluster because I have 9 tera of data so it can
    take one day to restart all cluster. In addition to that old machines and
    new machines will not know each other( I mean unicast variable of the
    elasticsearch.yml, the old machines will know just old ones). Do you think
    it can cause a problem ?

Thanks

29 Mayıs 2015 Cuma 14:45:40 UTC+2 tarihinde Nikolas Everett yazdı:

What you just described should work fine. exclude._ip will move the
shards off of the nodes you exclude but queries and updates can proceed
while this is happening because the data is still on the old nodes. The
updates will make their way to the new copies via a transaction log reply
mechanism.

Depending on how the clients connect you might need to shift their
connection uris to the new servers and that might require restarting them -
depending on how your clients work.

On Fri, May 29, 2015 at 8:38 AM, mehmet özer mehmeto...@gmail.com
wrote:

Hello,

I am running a cluster which contains many indexes. In the near future I
will have new machines so that I will need to migrate my indexes to those
machines. I have thought some scenarios but it will not be possible to do
so. Let me explain what I thought and why it was not possible;

  • I wanted to put the new machines into existing cluster and then I
    thought I could use exclude._ip option to exclude existing machines. So
    that old indexes would have migrated.

This seemed possible for me but I can not stop working on the index,
users should continue making searches on the indexes. So can you please
tell me how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/


You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0JOF3%2BAQ4Y0Pfsp8yWheec6WwnrKEuqjCPN7rPHqH24g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mehmet Özer) #5

Thats great. Thanks for your help !

29 Mayıs 2015 Cuma 15:09:21 UTC+2 tarihinde Nikolas Everett yazdı:

You certainly will have to think of a way to add new cluster masters - but
you can still do it with unicast discovery I think.

Yeah - exclude doesn't disable searches - it just causes elasticsearch to
move the shards using it normal shard moving mechanism. Its really just
adding a routing rule.

You shouldn't have to restart the whole cluster - but you will have to
convince your clients to look at the new nodes.

The last time I did a full restart of the Elasticsearch cluster was about
a year ago when I upgraded from 0.90.something to 1.1.something and I
expect the next full restart to be the upgrade to 2.0 when its released. In
the mean time we've:

  1. Done about twenty rolling restarts to pick up new code
  2. Replaced all the hard drives in all the nodes - two at a time - moving
    data off of the nodes we were restarting using excluse._ip
  3. Changed mount options on the elasticsearch data partition using the
    same technique
  4. More than doubled the number of nodes in the cluster.

In our case the application connects using linux virtual server so all we
had to do is teach the application to retry on connection errors for when
lvs round-robin-ed it to a down Elasticsearch node and teach our lvs pool
maintainer to depool elasticsearch nodes that are down.

I'm not an expert on the Java client so I don't know what you'd have to do
to update and java applications to the new nodes. If you are using a proxy
like nginx you'd just have to change where it points.

Nik

On Fri, May 29, 2015 at 8:51 AM, mehmet özer <mehmeto...@gmail.com
<javascript:>> wrote:

Thanks for your answer. I didnt know the exclude doesnt disable the
research. But according to the restart,

  • I cannot restart the cluster because I have 9 tera of data so it can
    take one day to restart all cluster. In addition to that old machines and
    new machines will not know each other( I mean unicast variable of the
    elasticsearch.yml, the old machines will know just old ones). Do you think
    it can cause a problem ?

Thanks

29 Mayıs 2015 Cuma 14:45:40 UTC+2 tarihinde Nikolas Everett yazdı:

What you just described should work fine. exclude._ip will move the
shards off of the nodes you exclude but queries and updates can proceed
while this is happening because the data is still on the old nodes. The
updates will make their way to the new copies via a transaction log reply
mechanism.

Depending on how the clients connect you might need to shift their
connection uris to the new servers and that might require restarting them -
depending on how your clients work.

On Fri, May 29, 2015 at 8:38 AM, mehmet özer mehmeto...@gmail.com
wrote:

Hello,

I am running a cluster which contains many indexes. In the near future
I will have new machines so that I will need to migrate my indexes to those
machines. I have thought some scenarios but it will not be possible to do
so. Let me explain what I thought and why it was not possible;

  • I wanted to put the new machines into existing cluster and then I
    thought I could use exclude._ip option to exclude existing machines. So
    that old indexes would have migrated.

This seemed possible for me but I can not stop working on the index,
users should continue making searches on the indexes. So can you please
tell me how I can migrate the data without disturbing clients ?

Thank you

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3df7a8b2-c231-4b27-b072-c4c8e790cd3c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/


You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c8cc4028-9ca6-4ffa-ad92-d36082062839%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0f797fb5-e8ef-4a34-9513-6ae891435976%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #6