The number of open files does not depend on the number of documents.
A shard comes not for free. Each shard can take around ~150 open file
descriptors (sockets, segment files) and up to 400-500 if actively being
indexed.
Take care of number of shards, if you have 5 shards per index, and 2000
indices per node, you would hvae to prepare 10k * 150 open file
descriptors. That is a challenge on a single RHEL 7 system providing 131072
file descriptors by default so you would have to change system limits (cat
/proc/sys/fs/file-max) - the default is already very high.
I recommend using fewer shards and redesign the application for fewer
indices (or even a single index) if you are limited to 2 nodes only. You
can look at shard routing and index aliasing if this helps:
Jörg
On Fri, May 1, 2015 at 5:05 PM, Ann Yablunovskaya lad.shada@gmail.com
wrote:
I am looking for suggestions on cluster configuration.
I have 2 nodes (master/data and data), 544 indices, about 800 mil
documents.
If I try to insert more documents and create more indices, I will
catch error "too many open files".
My node's configuration:
CentOS 7
Intel(R) Xeon(R) CPU x16
RAM 62 Gb
ulimit -n
100000
In future I will have a lot of indices (about 2000) and a lot of documents
(~5 bil or maybe more)
How can I avoid the error "too many open files"?
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_EjkMHgT_M_KPvV%3DDSdf-NyidqOziZvg5HXizx8J8rQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
