Maximum number of indices per machine


(adam.porat@gmail.com) #1

Hi,

  How many *indices per machine* (roughly) can ElasticSearch handle 

smoothly? Let's say average # of documents per index is 20 thousand, and
the machine is a powerful server.

Thank you.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c75e845a-9c95-498f-ba8e-9fe31320ad8b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #2

Can you be more specific about the definition of a "powerful" server?

Note, there are also different numbers for different workloads (just
indexing few docs per sec, bulk indexing, just a few searches per sec, huge
search load etc) so it really depends.

Jörg

On Mon, Mar 3, 2014 at 9:47 AM, Adam Porat adam.porat@gmail.com wrote:

Hi,

  How many *indices per machine* (roughly) can ElasticSearch handle

smoothly? Let's say average # of documents per index is 20 thousand, and
the machine is a powerful server.

Thank you.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c75e845a-9c95-498f-ba8e-9fe31320ad8b%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_w45nvNG3rGG3NtjS2uw41wcyOTwvtqtSHmz5vfV%2BdA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Itamar Syn-Hershko) #3

Joerg, I think what Adam means is how many open indexes an ES instance can
hold before lagging or crashing (assuming max open files is set correctly
etc).

Small indexes usually mean less segments per index, so the question may
boil down mostly to number of open IR/IW?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Mon, Mar 3, 2014 at 10:50 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Can you be more specific about the definition of a "powerful" server?

Note, there are also different numbers for different workloads (just
indexing few docs per sec, bulk indexing, just a few searches per sec, huge
search load etc) so it really depends.

Jörg

On Mon, Mar 3, 2014 at 9:47 AM, Adam Porat adam.porat@gmail.com wrote:

Hi,

  How many *indices per machine* (roughly) can ElasticSearch handle

smoothly? Let's say average # of documents per index is 20 thousand, and
the machine is a powerful server.

Thank you.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c75e845a-9c95-498f-ba8e-9fe31320ad8b%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_w45nvNG3rGG3NtjS2uw41wcyOTwvtqtSHmz5vfV%2BdA%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvKtLYz%2B_799y8Lqs9SbMJ7jbsftoyS783f308ibj-%2B1w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #4

Sure, you can create hundreds of thousands of indices, without docs, or
with just one doc. Closing an index even frees the resources of the index
management. This is not useful, just an edge case.

I'm afraid this is not the answer to the question.

The number of open indices (shards) is limited by disk space and maximum
number of file descriptors. But that is also theoretical. Using all indices
at once will use more resources than file descriptors. It depends on the
index / query characteristics (RAM for shards, fields, term count, term
distribution etc.) That is not directly related to "powerful" servers. Even
the weakest server can create as much indices as the strongest one.
Putting workload on indices is a different story.

Jörg

On Mon, Mar 3, 2014 at 11:21 AM, Itamar Syn-Hershko itamar@code972.comwrote:

Joerg, I think what Adam means is how many open indexes an ES instance can
hold before lagging or crashing (assuming max open files is set correctly
etc).

Small indexes usually mean less segments per index, so the question may
boil down mostly to number of open IR/IW?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE306tG3yRkCZkFTBmKRXmEWcOHyz3m9TgyT5Cmh03GwA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(adam.porat@gmail.com) #5

Hi Joerg & Itamar,
Thanks for your reply. What I am trying to get is an order of magnitue.
Suppose a server has 20GB RAM. Each index has about 20K documents, each
document is about 500 words, average term distribution. The question is
whether the server would be able to smoothly handle tenths, hundreds, or
thousands (or more?) of open indices. Suppose each index gets about 10 new
documents per minute, and serves about 30 queries per minute.
Thank you!

בתאריך יום שני, 3 במרץ 2014 19:04:22 UTC+2, מאת Jörg Prante:

Sure, you can create hundreds of thousands of indices, without docs, or
with just one doc. Closing an index even frees the resources of the index
management. This is not useful, just an edge case.

I'm afraid this is not the answer to the question.

The number of open indices (shards) is limited by disk space and maximum
number of file descriptors. But that is also theoretical. Using all indices
at once will use more resources than file descriptors. It depends on the
index / query characteristics (RAM for shards, fields, term count, term
distribution etc.) That is not directly related to "powerful" servers. Even
the weakest server can create as much indices as the strongest one.
Putting workload on indices is a different story.

Jörg

On Mon, Mar 3, 2014 at 11:21 AM, Itamar Syn-Hershko <ita...@code972.com<javascript:>

wrote:

Joerg, I think what Adam means is how many open indexes an ES instance
can hold before lagging or crashing (assuming max open files is set
correctly etc).

Small indexes usually mean less segments per index, so the question may
boil down mostly to number of open IR/IW?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/46fb6eed-ef7d-4fb1-93e7-020b0218c063%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #6