Too many open files

Franz_Allan_Valencia · July 16, 2010, 6:51am

I'm currently trying to index around 6.5M records (using java api, and using
the default operationThreaded which I believe is false). And I am sometimes
getting a (Too many open files).

What can I do to prevent this?

2010-07-16 15:36:29.617::WARN: Nested in

org.elasticsearch.transport.RemoteTransportException:
[Unuscione][inet[/10.0.6.1:9300]][indices/index/shard/index]:
java.io.FileNotFoundException:
/opt/elasticsearch/alternatives/elasticsearch-0.8.0/work/elasticsearch/indices/58e9d8b9-c42b-4261-a857-1d70d912dd2d/application/1/index/_2z6.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:65)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2089)
at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:204)
at
org.elasticsearch.index.shard.service.InternalIndexShard.innerIndex(InternalIndexShard.java:240)
at
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:228)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:125)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:56)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:328)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.access$400(TransportShardReplicationOperationAction.java:198)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:252)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

Thanks,

--
Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see

Franz_Allan_Valencia · July 16, 2010, 7:44am

Oh. And once that happens, I get very few successful shards, and a whole lot
of BroadcastShardOperationFailedException (which I can't seem to recover
from except to do the whole re-indexing again).

On Fri, Jul 16, 2010 at 2:51 PM, Franz Allan Valencia See <
franz.see@gmail.com> wrote:

I'm currently trying to index around 6.5M records (using java api, and
using the default operationThreaded which I believe is false). And I am
sometimes getting a (Too many open files).

What can I do to prevent this?

2010-07-16 15:36:29.617::WARN: Nested in

org.elasticsearch.transport.RemoteTransportException:
[Unuscione][inet[/10.0.6.1:9300]][indices/index/shard/index]:
java.io.FileNotFoundException:
/opt/elasticsearch/alternatives/elasticsearch-0.8.0/work/elasticsearch/indices/58e9d8b9-c42b-4261-a857-1d70d912dd2d/application/1/index/_2z6.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:65)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2089)
at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:204)
at
org.elasticsearch.index.shard.service.InternalIndexShard.innerIndex(InternalIndexShard.java:240)
at
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:228)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:125)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:56)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:328)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.access$400(TransportShardReplicationOperationAction.java:198)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:252)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

Thanks,

--
Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see

--
Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see

Clinton_Gormley · July 16, 2010, 9:19am

On Fri, 2010-07-16 at 15:44 +0800, Franz Allan Valencia See wrote:

Oh. And once that happens, I get very few successful shards, and a
whole lot of BroadcastShardOperationFailedException (which I can't
seem to recover from except to do the whole re-indexing again).

Set your ulimit high for whatever user is running elasticsearch, eg

ulimit -n 20000; ./bin/elasticsearch

To make it more permanent:

I have created a user called elasticsearch
and I added this entry to ï»¿/etc/security/limits.conf:

elasticsearch soft nofile 20000

clint

On Fri, Jul 16, 2010 at 2:51 PM, Franz Allan Valencia See
franz.see@gmail.com wrote:
I'm currently trying to index around 6.5M records (using java
api, and using the default operationThreaded which I believe
is false). And I am sometimes getting a (Too many open files).

    What can I do to prevent this?
    
            2010-07-16 15:36:29.617::WARN:  Nested in
            org.elasticsearch.transport.RemoteTransportException:
            [Unuscione][inet[/10.0.6.1:9300]][indices/index/shard/index]:
            java.io.FileNotFoundException: /opt/elasticsearch/alternatives/elasticsearch-0.8.0/work/elasticsearch/indices/58e9d8b9-c42b-4261-a857-1d70d912dd2d/application/1/index/_2z6.fdt (Too many open files)
                at java.io.RandomAccessFile.open(Native Method)
                at
            java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
                at org.apache.lucene.store.SimpleFSDirectory
            $SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180)
                at
            org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74)
                at
            org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:65)
                at
            org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
                at
            org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
                at org.apache.lucene.index.StoredFieldsWriter
            $PerDoc.finish(StoredFieldsWriter.java:193)
                at org.apache.lucene.index.DocumentsWriter
            $WaitQueue.writeDocument(DocumentsWriter.java:1460)
                at org.apache.lucene.index.DocumentsWriter
            $WaitQueue.add(DocumentsWriter.java:1479)
                at
            org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099)
                at
            org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777)
                at
            org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
                at
            org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2089)
                at
            org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:204)
                at
            org.elasticsearch.index.shard.service.InternalIndexShard.innerIndex(InternalIndexShard.java:240)
                at
            org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:228)
                at
            org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:125)
                at
            org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:56)
                at
            org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:328)
                at
            org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.access$400(TransportShardReplicationOperationAction.java:198)
                at
            org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:252)
                at java.util.concurrent.ThreadPoolExecutor
            $Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor
            $Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
    
    
    Thanks,
    
    -- 
    Franz Allan Valencia See | Java Software Engineer 
    franz.see@gmail.com
    LinkedIn: http://www.linkedin.com/in/franzsee
    Twitter: http://www.twitter.com/franz_see

--
Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see
--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.

Topic		Replies	Views
Getting Error after hitting to Elasticsearch through Java API(Too many open files) Elasticsearch	2	563	March 28, 2017
Too Many Open Files Elasticsearch	4	1582	July 6, 2017
Strange no. of file descriptors eating up Elasticsearch	14	794	July 6, 2017
Too many files open Elasticsearch	1	301	July 6, 2017
Too many open files in Elasticsearch Elasticsearch	4	2034	July 5, 2017

Too many open files

Related topics