Elasticsearch snapshots throttle problems

Elastic Stack Elasticsearch
1

Hi,

I have 12 elasticsearch nodes, with 10gb eth

Ive been having alot of problem with the performance of snapshots, its
throttles to 20 mb/s even tho i set max_snapshot_bytes_per_sec to something
else, ive tried to set it in bytes, in megabytes (500m, 500mb)

Ive tried to move 100gb file from elastic-node to my nfs-server, its about
1gb/s
Ive tried to move 100gb file from elastic-node down to my share, its about
1gb/s
Ive tried to just cp -rp my index from elastic-node to my share, its about
1gb/s

Am i missing something here? How is the max_snapshot_bytes_per_sec suppose
to look like?
Are there any other settings (like recovery streams etc) that affects this?

My backup dir:
curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
"type": "fs",
"settings": {
"location": "/misc/backup_elastic/snapshot",
"compress": true,
"verify": true,
"max_snapshot_bytes_per_sec" : "1048576000",
"max_restore_bytes_per_sec" : "1048576000"
}
}'

And snapshot:
curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_test" -d '{
"indices": "index-2014-01-03",
"ignore_unavailable": "true",
"include_global_state": "true",
"partial": "true"
}'

Regards, Johan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fcb8e013-6174-44f7-9343-5090665c43f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I noticed these warnings on some of my nodes while executing the snapshot,
maybe it has to do something with why its so slow.

[2014-12-03 15:57:35,699][WARN ][snapshots ] [xx06]
[[xxx-2014-11-20][7]] [my_backup:snapshot_test] failed to create snapshot

org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException:
[xxx-2014-11-20][7] Aborted

at
org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext$AbortableInputStream.checkAborted(BlobStoreIndexShardRepository.java:632)

at
org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext$AbortableInputStream.read(BlobStoreIndexShardRepository.java:625)

at java.io.FilterInputStream.read(FilterInputStream.java:107)

at
org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshotFile(BlobStoreIndexShardRepository.java:557)

at
org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:501)

at
org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:139)

at
org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:86)

at
org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:818)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745

Den onsdagen den 3:e december 2014 kl. 10:25:38 UTC+1 skrev Johan Öhr:

Hi,

I have 12 elasticsearch nodes, with 10gb eth

Ive been having alot of problem with the performance of snapshots, its
throttles to 20 mb/s even tho i set max_snapshot_bytes_per_sec to something
else, ive tried to set it in bytes, in megabytes (500m, 500mb)

Ive tried to move 100gb file from elastic-node to my nfs-server, its about
1gb/s
Ive tried to move 100gb file from elastic-node down to my share, its about
1gb/s
Ive tried to just cp -rp my index from elastic-node to my share, its about
1gb/s

Am i missing something here? How is the max_snapshot_bytes_per_sec suppose
to look like?
Are there any other settings (like recovery streams etc) that affects
this?

My backup dir:
curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
"type": "fs",
"settings": {
"location": "/misc/backup_elastic/snapshot",
"compress": true,
"verify": true,
"max_snapshot_bytes_per_sec" : "1048576000",
"max_restore_bytes_per_sec" : "1048576000"
}
}'

And snapshot:
curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_test" -d '{
"indices": "index-2014-01-03",
"ignore_unavailable": "true",
"include_global_state": "true",
"partial": "true"
}'

Regards, Johan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13857ee7-5e36-45a5-a2b5-3ffa66e8fa21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Those settings look correct to me. You can set using kb,mb,gb, etc.

On Wednesday, December 3, 2014 4:25:38 AM UTC-5, Johan Öhr wrote:

Hi,

I have 12 elasticsearch nodes, with 10gb eth

Ive been having alot of problem with the performance of snapshots, its
throttles to 20 mb/s even tho i set max_snapshot_bytes_per_sec to something
else, ive tried to set it in bytes, in megabytes (500m, 500mb)

Ive tried to move 100gb file from elastic-node to my nfs-server, its about
1gb/s
Ive tried to move 100gb file from elastic-node down to my share, its about
1gb/s
Ive tried to just cp -rp my index from elastic-node to my share, its about
1gb/s

Am i missing something here? How is the max_snapshot_bytes_per_sec suppose
to look like?
Are there any other settings (like recovery streams etc) that affects
this?

My backup dir:
curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
"type": "fs",
"settings": {
"location": "/misc/backup_elastic/snapshot",
"compress": true,
"verify": true,
"max_snapshot_bytes_per_sec" : "1048576000",
"max_restore_bytes_per_sec" : "1048576000"
}
}'

And snapshot:
curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_test" -d '{
"indices": "index-2014-01-03",
"ignore_unavailable": "true",
"include_global_state": "true",
"partial": "true"
}'

Regards, Johan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5b753ce0-42d1-4b47-aab4-4d021bcaf555%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.