Index reallocation takes hours

Hey guys,

we have setup an central ES-server with 2 instances (1 instance SSDs for
indexing/last indices and an other one for long term storage)

Everyday at 6 a.m a cronjob is started that moves indices older than 7 days
to the HDD storage node.

Our indices have an average size of 300 GB, but it takes about 6-7 hours
till the reallocation of 1 index is done (if you copy it manually with "cp"
it only takes about 30 minutes).

Is there any settings which can i adjust that speed this process of
reallocation up?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aa78aefb-be9e-4dc6-bdca-948d312f06af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Currently shard allocation is throttled by default with a target low value.
If you are using an older version of Elasticsearch, then a Lucene bug will
make it even slower.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html#store-throttling

How many shards do your indices have? The default number of concurrent
allocations is only 2. The ideal values depends on your hardware,
especially your disk io.

Cheers,
Ivan
On Oct 7, 2014 8:42 AM, "Andrew Lakes" alakes501@gmail.com wrote:

Hey guys,

we have setup an central ES-server with 2 instances (1 instance SSDs for
indexing/last indices and an other one for long term storage)

Everyday at 6 a.m a cronjob is started that moves indices older than 7
days to the HDD storage node.

Our indices have an average size of 300 GB, but it takes about 6-7 hours
till the reallocation of 1 index is done (if you copy it manually with "cp"
it only takes about 30 minutes).

Is there any settings which can i adjust that speed this process of
reallocation up?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aa78aefb-be9e-4dc6-bdca-948d312f06af%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/aa78aefb-be9e-4dc6-bdca-948d312f06af%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAdR_Fj_kDu7vtg6pc5_FkQ7A1UF6jR6tT56tVbPcxbsw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

we are using version 1.3.2 of ES , each index have 4 shards.

I will try adjusting the store-throttling to an acceptable number without
crushing our disk IOs

Am Dienstag, 7. Oktober 2014 19:35:35 UTC+2 schrieb Ivan Brusic:

Currently shard allocation is throttled by default with a target low
value. If you are using an older version of Elasticsearch, then a Lucene
bug will make it even slower.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html#store-throttling

How many shards do your indices have? The default number of concurrent
allocations is only 2. The ideal values depends on your hardware,
especially your disk io.

Cheers,
Ivan
On Oct 7, 2014 8:42 AM, "Andrew Lakes" <alak...@gmail.com <javascript:>>
wrote:

Hey guys,

we have setup an central ES-server with 2 instances (1 instance SSDs for
indexing/last indices and an other one for long term storage)

Everyday at 6 a.m a cronjob is started that moves indices older than 7
days to the HDD storage node.

Our indices have an average size of 300 GB, but it takes about 6-7 hours
till the reallocation of 1 index is done (if you copy it manually with "cp"
it only takes about 30 minutes).

Is there any settings which can i adjust that speed this process of
reallocation up?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aa78aefb-be9e-4dc6-bdca-948d312f06af%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/aa78aefb-be9e-4dc6-bdca-948d312f06af%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/09f3f2db-0ae7-4f0a-8891-e4ab267d1c78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi,

i want to use this thread, to clarify some of the elasticsearch settings,
because im struggeling a bit in understanding all of them.

1.indices.store.throttle.max_bytes_per_sec : This is the maximum amount of
mb/gb which elasticsearch uses per second to to the merging process -
right? (Is there an difference between
"indices.store.throttle.max_bytes_per_sec" and
"index.store.throttle.max_bytes_per_sec"? )
2.indices.recovery.max_bytes_per_sec: This is the maximum amout of mb/gb
which elasticsearch uses to initialize shards/indices after an restart of
the node - right?
3.indices.recovery.concurrent_streams: The amount of parallel shard
initializations after an restart - right?
4.cluster.routing.allocation.node_concurrent_recoveries: The cluster wide
amount of parallel shard initializations after an restart - right?
5.cluster.routing.allocation.node_initial_primaries_recoveries: The cluster
wide amount of parallel primary shard initializations after an restart -
right?
6.cluster.routing.allocation.cluster_concurrent_rebalance:2: The amount of
shards that are parallel allowed to move from one node to an other -right?

But of all the settings found in the elasticsearch there is no such
settings like "cluster.routing.allocation.max_bytes_per_sec" which limits
the maximum amount of mb/gb per sec that can be used to allocate a shard
from one node to another - will elasticsearch by default use a value or
just use the maximum possible speed?

Thanks for response

Am Mittwoch, 8. Oktober 2014 11:04:26 UTC+2 schrieb Andrew Lakes:

Hi,

we are using version 1.3.2 of ES , each index have 4 shards.

I will try adjusting the store-throttling to an acceptable number without
crushing our disk IOs

Am Dienstag, 7. Oktober 2014 19:35:35 UTC+2 schrieb Ivan Brusic:

Currently shard allocation is throttled by default with a target low
value. If you are using an older version of Elasticsearch, then a Lucene
bug will make it even slower.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html#store-throttling

How many shards do your indices have? The default number of concurrent
allocations is only 2. The ideal values depends on your hardware,
especially your disk io.

Cheers,
Ivan
On Oct 7, 2014 8:42 AM, "Andrew Lakes" alak...@gmail.com wrote:

Hey guys,

we have setup an central ES-server with 2 instances (1 instance SSDs for
indexing/last indices and an other one for long term storage)

Everyday at 6 a.m a cronjob is started that moves indices older than 7
days to the HDD storage node.

Our indices have an average size of 300 GB, but it takes about 6-7 hours
till the reallocation of 1 index is done (if you copy it manually with "cp"
it only takes about 30 minutes).

Is there any settings which can i adjust that speed this process of
reallocation up?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aa78aefb-be9e-4dc6-bdca-948d312f06af%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/aa78aefb-be9e-4dc6-bdca-948d312f06af%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1224c3f7-8207-42f5-8d2d-95eb36280ef2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.