Problem with file descriptors

Rino_Rondan · February 10, 2014, 2:01pm

Hi:

I want to know why or how works ES, because it needs a lot of file
descriptors, when i doing re indexing it uses a lot. But if someone can
explain me or share some links to understand why open a lot of files and if
it degrade the performance of servers.

Regards

ps: I change on sysctl.conf, and it works, but i want to know why i need to
do that.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/640dc816-1686-465a-8f5d-1a988c7ebb8b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Binh_Ly · February 10, 2014, 2:22pm

This might help:

http://lucene.apache.org/core/3_0_3/fileformats.html#Index%20File%20Formats

Keep in mind, each shard in ES is a Lucene index so that is while lots of
file descriptors need to be used. In other words, it's normal.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e6c59871-6c1f-45c0-8060-e911a2c0609f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Rino_Rondan · February 10, 2014, 6:38pm

Thanks Binh, I have another question what is the best way to delete an
index and the shards that use it to then recreate this index from beginning.

Regards

On Mon, Feb 10, 2014 at 11:22 AM, Binh Ly binh@hibalo.com wrote:

This might help:

Apache Lucene - Index File Formats

Keep in mind, each shard in ES is a Lucene index so that is while lots of
file descriptors need to be used. In other words, it's normal.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/nadGbpBPAQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e6c59871-6c1f-45c0-8060-e911a2c0609f%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--

Rondan Rino
Certificado en LPIC-2 https://cs.lpi.org/caf/Xamman/certification
LPI ID:LPI000209832
Verification Code:gbblvwyfxu
Red Hat Certified Engineer -- RHCE --
RHCVAhttps://www.redhat.com/wapps/training/certification/verify.html?certNumber=120-175-658&verify=Verify

Blog:http://www.itrestauracion.com.ar
Cv: http://cv.rinorondan.com.ar http://www.rinorondan.com.ar/
http://counter.li.org Linux User -> #517918
Viva La Santa Federacion!!
Mueran Los Salvages Unitarios!!
^^^Transcripcion de la epoca ^^^

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAASrp47_ie1YHbHgS2Tmo70sFN%2BXv4%3D%2B-nS_fY5QJu1zOd8JJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Binh_Ly · February 10, 2014, 6:43pm

Rondan,

If you're using REST/HTTP, you can use the Delete Index API to delete an
index easily:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-delete-index.html

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b2421e6b-3114-4ec8-b417-fd594d7e2612%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Rino_Rondan · February 10, 2014, 6:57pm

THanks, yes we are using this but with java api, but exists any formula
that i can predict how many open file i will have if my indices increase
every week, in order to configure the correct number of file descriptors.
Because when i delete an index and it will create again all (index and all
shards for this index), the quantity of file descriptors is more than that
i have all index working fine.(without delete all)

Regards

On Mon, Feb 10, 2014 at 3:43 PM, Binh Ly binh@hibalo.com wrote:

Rondan,

If you're using REST/HTTP, you can use the Delete Index API to delete an
index easily:

Elasticsearch Platform — Find real-time answers at scale | Elastic

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/nadGbpBPAQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b2421e6b-3114-4ec8-b417-fd594d7e2612%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--

Rondan Rino
Certificado en LPIC-2 https://cs.lpi.org/caf/Xamman/certification
LPI ID:LPI000209832
Verification Code:gbblvwyfxu
Red Hat Certified Engineer -- RHCE --
RHCVAhttps://www.redhat.com/wapps/training/certification/verify.html?certNumber=120-175-658&verify=Verify

Blog:http://www.itrestauracion.com.ar
Cv: http://cv.rinorondan.com.ar http://www.rinorondan.com.ar/
http://counter.li.org Linux User -> #517918
Viva La Santa Federacion!!
Mueran Los Salvages Unitarios!!
^^^Transcripcion de la epoca ^^^

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAASrp46X__L-MaMg1do%3DKNd1Zf6CLxz8vCUfVZubT8h_Uxxojg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Binh_Ly · February 10, 2014, 7:06pm

Rondan,

For Java, should be something like this:

      DeleteIndexResponse deleteIndexResponse = new

DeleteIndexRequestBuilder(client.admin().indices(),
INDEX_NAME).execute().actionGet();
if (deleteIndexResponse.isAcknowledged()) System.out.println(
"Deleted index" );

Generally, setting the file descriptors to 65535 should suffice. You can
learn more about this and other things here:

On Monday, February 10, 2014 1:57:51 PM UTC-5, Rino Rondan wrote:

THanks, yes we are using this but with java api, but exists any formula
that i can predict how many open file i will have if my indices increase
every week, in order to configure the correct number of file descriptors.
Because when i delete an index and it will create again all (index and all
shards for this index), the quantity of file descriptors is more than that
i have all index working fine.(without delete all)

Regards

On Mon, Feb 10, 2014 at 3:43 PM, Binh Ly <bi...@hibalo.com <javascript:>>wrote:

Rondan,

If you're using REST/HTTP, you can use the Delete Index API to delete an
index easily:

Elasticsearch Platform — Find real-time answers at scale | Elastic

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/nadGbpBPAQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b2421e6b-3114-4ec8-b417-fd594d7e2612%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--

Rondan Rino
Certificado en LPIC-2 https://cs.lpi.org/caf/Xamman/certification
LPI ID:LPI000209832
Verification Code:gbblvwyfxu
Red Hat Certified Engineer -- RHCE -- RHCVAhttps://www.redhat.com/wapps/training/certification/verify.html?certNumber=120-175-658&verify=Verify

Blog:http://www.itrestauracion.com.ar
Cv: http://cv.rinorondan.com.ar http://www.rinorondan.com.ar/
http://counter.li.org Linux User -> #517918
Viva La Santa Federacion!!
Mueran Los Salvages Unitarios!!
^^^Transcripcion de la epoca ^^^

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7a320663-8bab-4f67-b9a4-b8a4c0199823%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Rino_Rondan · February 10, 2014, 7:59pm

THanks a lot

On Mon, Feb 10, 2014 at 4:06 PM, Binh Ly binh@hibalo.com wrote:

Rondan,

For Java, should be something like this:
      DeleteIndexResponse deleteIndexResponse = new
DeleteIndexRequestBuilder(client.admin().indices(),
INDEX_NAME).execute().actionGet();
if (deleteIndexResponse.isAcknowledged()) System.out.println(
"Deleted index" );

Generally, setting the file descriptors to 65535 should suffice. You can
learn more about this and other things here:

Elasticsearch Platform — Find real-time answers at scale | Elastic

On Monday, February 10, 2014 1:57:51 PM UTC-5, Rino Rondan wrote:

THanks, yes we are using this but with java api, but exists any formula
that i can predict how many open file i will have if my indices increase
every week, in order to configure the correct number of file descriptors.
Because when i delete an index and it will create again all (index and all
shards for this index), the quantity of file descriptors is more than that
i have all index working fine.(without delete all)

Regards

On Mon, Feb 10, 2014 at 3:43 PM, Binh Ly bi...@hibalo.com wrote:

Rondan,

If you're using REST/HTTP, you can use the Delete Index API to delete an
index easily:

Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/indices-delete-index.html

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/elasticsearch/nadGbpBPAQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/b2421e6b-3114-4ec8-b417-fd594d7e2612%
40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Rondan Rino
Certificado en LPIC-2 https://cs.lpi.org/caf/Xamman/certification
LPI ID:LPI000209832
Verification Code:gbblvwyfxu
Red Hat Certified Engineer -- RHCE -- RHCVAhttps://www.redhat.com/wapps/training/certification/verify.html?certNumber=120-175-658&verify=Verify

Blog:http://www.itrestauracion.com.ar
Cv: http://cv.rinorondan.com.ar http://www.rinorondan.com.ar/
http://counter.li.org Linux User -> #517918
Viva La Santa Federacion!!
Mueran Los Salvages Unitarios!!
^^^Transcripcion de la epoca ^^^

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/nadGbpBPAQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7a320663-8bab-4f67-b9a4-b8a4c0199823%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--

Rondan Rino
Certificado en LPIC-2 https://cs.lpi.org/caf/Xamman/certification
LPI ID:LPI000209832
Verification Code:gbblvwyfxu
Red Hat Certified Engineer -- RHCE --
RHCVAhttps://www.redhat.com/wapps/training/certification/verify.html?certNumber=120-175-658&verify=Verify

Blog:http://www.itrestauracion.com.ar
Cv: http://cv.rinorondan.com.ar http://www.rinorondan.com.ar/
http://counter.li.org Linux User -> #517918
Viva La Santa Federacion!!
Mueran Los Salvages Unitarios!!
^^^Transcripcion de la epoca ^^^

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAASrp44TZFd0ibuFsvQ0uZvHqnsEh6J_jf5Y%3DhmJO6r-1eH9Rg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.