"too many open files" problems and suggestions on cluster configuration


(Ann Yablunovskaya) #1

I am looking for suggestions on cluster configuration.

I have 2 nodes (master/data and data), 544 indices, about 800 mil documents.

If I try to insert more documents and create more indices, I will
catch error "too many open files".

My node's configuration:

CentOS 7
Intel(R) Xeon(R) CPU x16
RAM 62 Gb

ulimit -n

100000

In future I will have a lot of indices (about 2000) and a lot of documents
(~5 bil or maybe more)

How can I avoid the error "too many open files"?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #2

Add more nodes or reduce the number of shards per node.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 mai 2015 à 17:05, Ann Yablunovskaya lad.shada@gmail.com a écrit :

I am looking for suggestions on cluster configuration.

I have 2 nodes (master/data and data), 544 indices, about 800 mil documents.

If I try to insert more documents and create more indices, I will catch error "too many open files".

My node's configuration:

CentOS 7
Intel(R) Xeon(R) CPU x16
RAM 62 Gb

ulimit -n

100000

In future I will have a lot of indices (about 2000) and a lot of documents (~5 bil or maybe more)

How can I avoid the error "too many open files"?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<es_config.pp>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54E58499-F862-4427-A765-E72FCBDC8D92%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(Ann Yablunovskaya) #3

How to calculate the best amount of shards?

пятница, 1 мая 2015 г., 18:21:47 UTC+3 пользователь David Pilato написал:

Add more nodes or reduce the number of shards per node.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 mai 2015 à 17:05, Ann Yablunovskaya <lad....@gmail.com <javascript:>>
a écrit :

I am looking for suggestions on cluster configuration.

I have 2 nodes (master/data and data), 544 indices, about 800 mil
documents.

If I try to insert more documents and create more indices, I will
catch error "too many open files".

My node's configuration:

CentOS 7
Intel(R) Xeon(R) CPU x16
RAM 62 Gb

ulimit -n

100000

In future I will have a lot of indices (about 2000) and a lot of documents
(~5 bil or maybe more)

How can I avoid the error "too many open files"?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

<es_config.pp>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7c2e1952-e718-4563-ac5c-bb92b45b0aa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #4

The number of open files does not depend on the number of documents.

A shard comes not for free. Each shard can take around ~150 open file
descriptors (sockets, segment files) and up to 400-500 if actively being
indexed.

Take care of number of shards, if you have 5 shards per index, and 2000
indices per node, you would hvae to prepare 10k * 150 open file
descriptors. That is a challenge on a single RHEL 7 system providing 131072
file descriptors by default so you would have to change system limits (cat
/proc/sys/fs/file-max) - the default is already very high.

I recommend using fewer shards and redesign the application for fewer
indices (or even a single index) if you are limited to 2 nodes only. You
can look at shard routing and index aliasing if this helps:

http://www.elastic.co/guide/en/elasticsearch/guide/master/kagillion-shards.html

http://www.elastic.co/guide/en/elasticsearch/guide/master/faking-it.html

Jörg

On Fri, May 1, 2015 at 5:05 PM, Ann Yablunovskaya lad.shada@gmail.com
wrote:

I am looking for suggestions on cluster configuration.

I have 2 nodes (master/data and data), 544 indices, about 800 mil
documents.

If I try to insert more documents and create more indices, I will
catch error "too many open files".

My node's configuration:

CentOS 7
Intel(R) Xeon(R) CPU x16
RAM 62 Gb

ulimit -n

100000

In future I will have a lot of indices (about 2000) and a lot of documents
(~5 bil or maybe more)

How can I avoid the error "too many open files"?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c5d45b95-b3d7-4b6a-80fa-111d66f3f65a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_EjkMHgT_M_KPvV%3DDSdf-NyidqOziZvg5HXizx8J8rQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #5