Slow cluster startup (again)

Elastic Stack Elasticsearch
1

Hello,

I have 3 nodes (latest ES 0.90.5).

index.number_of_shards = 5
index.number_of_replicas = 2

Gbit internet, SSD drives for /var/lib/elasticsearch.
I have about 1000 indexes created.
Total indexes size ~15GB.

How long should the startup of ES take ?
I keep having status red after hours.
unassigned_shards counter goes down really slow.

Currently we have one index per user. Would it help the startup time if we
make on BIG index and add user_id as search key ?
Any other tricks i can do to have a faster startup ?

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Elasticsearch has been throttling I/O recovery since version 0.90. The
defaults are fairly low. Trying increasing the
indices.recovery.max_bytes_per_sec setting. With SSD drives, the setting
could easily be over 100mb. Mileage may vary.

You can also increase the number of shards that are recovered at the same
time. The default is 2.

Cheers,

Ivan

On Mon, Oct 7, 2013 at 1:53 PM, Catalin Constantin dazoot@gmail.com wrote:

Hello,

I have 3 nodes (latest ES 0.90.5).

index.number_of_shards = 5
index.number_of_replicas = 2

Gbit internet, SSD drives for /var/lib/elasticsearch.
I have about 1000 indexes created.
Total indexes size ~15GB.

How long should the startup of ES take ?
I keep having status red after hours.
unassigned_shards counter goes down really slow.

Currently we have one index per user. Would it help the startup time if we
make on BIG index and add user_id as search key ?
Any other tricks i can do to have a faster startup ?

Thanks,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Yes, it will definitely help if you reduce the number of indices and use
aliases.

Jörg
Am 07.10.2013 22:53 schrieb "Catalin Constantin" dazoot@gmail.com:

Hello,

I have 3 nodes (latest ES 0.90.5).

index.number_of_shards = 5
index.number_of_replicas = 2

Gbit internet, SSD drives for /var/lib/elasticsearch.
I have about 1000 indexes created.
Total indexes size ~15GB.

How long should the startup of ES take ?
I keep having status red after hours.
unassigned_shards counter goes down really slow.

Currently we have one index per user. Would it help the startup time if we
make on BIG index and add user_id as search key ?
Any other tricks i can do to have a faster startup ?

Thanks,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

What about search speed ? Will it be affected ?
There will be a huge index with millions of documents.

Is this slow startup problem considered an issue ?
Will it be fixed in the future releases ?

While the cluster state is RED, no write operations can be made to it.
This means "downtime".

Thanks.

--
Catalin

On Tue, Oct 8, 2013 at 10:48 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Yes, it will definitely help if you reduce the number of indices and use
aliases.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

This happens because you decided to put hundreds of shards on a single node.

Look at node_concurrent_recoveries, the limiting factor so nodes can serve
requests, even if indices are recovering. If you set this value too high
you will get high iowait or timeouts.

You can look at your RAM and with 15G, the chance is high you can keep all
the index files in memory which is very fast.

Jörg
Am 08.10.2013 10:08 schrieb "Catalin Constantin" dazoot@gmail.com:

What about search speed ? Will it be affected ?
There will be a huge index with millions of documents.

Is this slow startup problem considered an issue ?
Will it be fixed in the future releases ?

While the cluster state is RED, no write operations can be made to it.
This means "downtime".

Thanks.

--
Catalin

On Tue, Oct 8, 2013 at 10:48 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Yes, it will definitely help if you reduce the number of indices and use
aliases.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.