Elastic search S3 backup speed

Elastic Stack Elasticsearch
1

Hello,

I am using "elasticsearch-cloud-aws" to create backups for my cluster in
S3, but the backups are taking too long to finish, by using cloud watch i
am seeing an average of of 760 MB/minute of network out in the machines, i
used to have similar numbers while the index was balancing itself but by
changing "indices.recovery.max_bytes_per_sec: 1000mb" i got much better
speeds, topping at 5GB/minute.
I would like to know if its there another property that maybe will make the
backup faster or its there a way to make this backups faster.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6b89b7ca-48a5-4820-840e-7ba5c7522087%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

You can set max_snapshot_bytes_per_sec -

There is also a snapshot threadpool you can experiment with, but it's not
recommended, I'm just making you aware of it -

But you are ultimately limited to your network speed.

On 16 March 2015 at 06:44, Alejandro Calderon <
alejandrocalderonvelez@gmail.com> wrote:

Hello,

I am using "elasticsearch-cloud-aws" to create backups for my cluster in
S3, but the backups are taking too long to finish, by using cloud watch i
am seeing an average of of 760 MB/minute of network out in the machines, i
used to have similar numbers while the index was balancing itself but by
changing "indices.recovery.max_bytes_per_sec: 1000mb" i got much better
speeds, topping at 5GB/minute.
I would like to know if its there another property that maybe will make
the backup faster or its there a way to make this backups faster.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6b89b7ca-48a5-4820-840e-7ba5c7522087%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6b89b7ca-48a5-4820-840e-7ba5c7522087%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_z_EfzLTiG%2BPmPJHz9w%3DKRSUdoGhwpGa_MUcQXx5u3RQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks i tested the settings and the speed get better, I also found out ES
1.4.0 has a bug about the snapshot/restore, after upgrading to 1.4.4 my
speed triplicated.

On Monday, March 16, 2015 at 8:44:32 AM UTC-5, Alejandro Calderon wrote:

Hello,

I am using "elasticsearch-cloud-aws" to create backups for my cluster in
S3, but the backups are taking too long to finish, by using cloud watch i
am seeing an average of of 760 MB/minute of network out in the machines, i
used to have similar numbers while the index was balancing itself but by
changing "indices.recovery.max_bytes_per_sec: 1000mb" i got much better
speeds, topping at 5GB/minute.
I would like to know if its there another property that maybe will make
the backup faster or its there a way to make this backups faster.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c9e7772b-2980-48f3-934c-cb02c9a105ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.