Powerful cluster is not able to handle 1.5Tb of data, how to optimize?

Thanks Greg,

Your example should be useful.

среда, 17 сентября 2014 г. пользователь Greg Murnane написал:

I run 1.3TB of active indices on a single node (64 GB ram with 12GB heap
size, and 15 small disks in a raid 5), with most of my messages quite
small, which makes it looks similar to your case in volume, although I have
a significantly lower (about 5K) indexing rate.

I suspect that the single disk on each node may be limiting your capacity
here. Try measuring some metrics of disk utilization (read and write rates,
latency, and queue lengths). If those all look fine while you're loading
the dashboard, then I'm incorrect. If they look like the disk is being
thrashed, consider moving to more smaller nodes, or adding additional disks
to these nodes to boost the IO rates.

The information transmitted in this email is intended only for the
person(s) or entity to which it is addressed and may contain confidential
and/or privileged material. Any review, retransmission, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient is prohibited. If you
received this email in error, please contact the sender and permanently
delete the email from any computer.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/4eP-aR0KpBA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','elasticsearch%2Bunsubscribe@googlegroups.com');>
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cc91b9f2-72b7-455d-b2d3-a1018d9ce919%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cc91b9f2-72b7-455d-b2d3-a1018d9ce919%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
Kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFVUaqPyh8C7QQW0n4M-1jPuyQn_vHXpb64%3DaZYhOgj%2BbD0gXw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.