Elastic search S3 backup speed

Hello,

I am using "elasticsearch-cloud-aws" to create backups for my cluster in
S3, but the backups are taking too long to finish, by using cloud watch i
am seeing an average of of 760 MB/minute of network out in the machines, i
used to have similar numbers while the index was balancing itself but by
changing "indices.recovery.max_bytes_per_sec: 1000mb" i got much better
speeds, topping at 5GB/minute.
I would like to know if its there another property that maybe will make the
backup faster or its there a way to make this backups faster.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6b89b7ca-48a5-4820-840e-7ba5c7522087%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You can set max_snapshot_bytes_per_sec -
http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#_common_repository_settings

There is also a snapshot threadpool you can experiment with, but it's not
recommended, I'm just making you aware of it -
http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html#modules-threadpool

But you are ultimately limited to your network speed.

On 16 March 2015 at 06:44, Alejandro Calderon <
alejandrocalderonvelez@gmail.com> wrote:

Hello,

I am using "elasticsearch-cloud-aws" to create backups for my cluster in
S3, but the backups are taking too long to finish, by using cloud watch i
am seeing an average of of 760 MB/minute of network out in the machines, i
used to have similar numbers while the index was balancing itself but by
changing "indices.recovery.max_bytes_per_sec: 1000mb" i got much better
speeds, topping at 5GB/minute.
I would like to know if its there another property that maybe will make
the backup faster or its there a way to make this backups faster.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6b89b7ca-48a5-4820-840e-7ba5c7522087%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6b89b7ca-48a5-4820-840e-7ba5c7522087%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_z_EfzLTiG%2BPmPJHz9w%3DKRSUdoGhwpGa_MUcQXx5u3RQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thanks i tested the settings and the speed get better, I also found out ES
1.4.0 has a bug about the snapshot/restore, after upgrading to 1.4.4 my
speed triplicated.

On Monday, March 16, 2015 at 8:44:32 AM UTC-5, Alejandro Calderon wrote:

Hello,

I am using "elasticsearch-cloud-aws" to create backups for my cluster in
S3, but the backups are taking too long to finish, by using cloud watch i
am seeing an average of of 760 MB/minute of network out in the machines, i
used to have similar numbers while the index was balancing itself but by
changing "indices.recovery.max_bytes_per_sec: 1000mb" i got much better
speeds, topping at 5GB/minute.
I would like to know if its there another property that maybe will make
the backup faster or its there a way to make this backups faster.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c9e7772b-2980-48f3-934c-cb02c9a105ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.