Too many open files


(Franz Allan Valencia See) #1

I'm currently trying to index around 6.5M records (using java api, and using
the default operationThreaded which I believe is false). And I am sometimes
getting a (Too many open files).

What can I do to prevent this?

2010-07-16 15:36:29.617::WARN: Nested in

org.elasticsearch.transport.RemoteTransportException:
[Unuscione][inet[/10.0.6.1:9300]][indices/index/shard/index]:
java.io.FileNotFoundException:
/opt/elasticsearch/alternatives/elasticsearch-0.8.0/work/elasticsearch/indices/58e9d8b9-c42b-4261-a857-1d70d912dd2d/application/1/index/_2z6.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:65)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2089)
at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:204)
at
org.elasticsearch.index.shard.service.InternalIndexShard.innerIndex(InternalIndexShard.java:240)
at
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:228)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:125)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:56)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:328)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.access$400(TransportShardReplicationOperationAction.java:198)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:252)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

Thanks,

--
Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see


(Franz Allan Valencia See) #2

Oh. And once that happens, I get very few successful shards, and a whole lot
of BroadcastShardOperationFailedException (which I can't seem to recover
from except to do the whole re-indexing again).

On Fri, Jul 16, 2010 at 2:51 PM, Franz Allan Valencia See <
franz.see@gmail.com> wrote:

I'm currently trying to index around 6.5M records (using java api, and
using the default operationThreaded which I believe is false). And I am
sometimes getting a (Too many open files).

What can I do to prevent this?

2010-07-16 15:36:29.617::WARN: Nested in

org.elasticsearch.transport.RemoteTransportException:
[Unuscione][inet[/10.0.6.1:9300]][indices/index/shard/index]:
java.io.FileNotFoundException:
/opt/elasticsearch/alternatives/elasticsearch-0.8.0/work/elasticsearch/indices/58e9d8b9-c42b-4261-a857-1d70d912dd2d/application/1/index/_2z6.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:180)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74)
at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:65)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:193)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1460)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1479)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2089)
at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:204)
at
org.elasticsearch.index.shard.service.InternalIndexShard.innerIndex(InternalIndexShard.java:240)
at
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:228)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:125)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:56)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:328)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.access$400(TransportShardReplicationOperationAction.java:198)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:252)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

Thanks,

--
Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see

--
Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see


(Clinton Gormley) #3

On Fri, 2010-07-16 at 15:44 +0800, Franz Allan Valencia See wrote:

Oh. And once that happens, I get very few successful shards, and a
whole lot of BroadcastShardOperationFailedException (which I can't
seem to recover from except to do the whole re-indexing again).

Set your ulimit high for whatever user is running elasticsearch, eg

ulimit -n 20000; ./bin/elasticsearch

To make it more permanent:

  • I have created a user called elasticsearch

  • and I added this entry to /etc/security/limits.conf:

    elasticsearch soft nofile 20000

clint

On Fri, Jul 16, 2010 at 2:51 PM, Franz Allan Valencia See
franz.see@gmail.com wrote:
I'm currently trying to index around 6.5M records (using java
api, and using the default operationThreaded which I believe
is false). And I am sometimes getting a (Too many open files).

    What can I do to prevent this?
    
            2010-07-16 15:36:29.617::WARN:  Nested in
            org.elasticsearch.transport.RemoteTransportException:
            [Unuscione][inet[/10.0.6.1:9300]][indices/index/shard/index]:
            java.io.FileNotFoundException: /opt/elasticsearch/alternatives/elasticsearch-0.8.0/work/elasticsearch/indices/58e9d8b9-c42b-4261-a857-1d70d912dd2d/application/1/index/_2z6.fdt (Too many open files)
                at java.io.RandomAccessFile.open(Native Method)
                at
            java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
                at org.apache.lucene.store.SimpleFSDirectory
            $SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180)
                at
            org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74)
                at
            org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:65)
                at
            org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:66)
                at
            org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:144)
                at org.apache.lucene.index.StoredFieldsWriter
            $PerDoc.finish(StoredFieldsWriter.java:193)
                at org.apache.lucene.index.DocumentsWriter
            $WaitQueue.writeDocument(DocumentsWriter.java:1460)
                at org.apache.lucene.index.DocumentsWriter
            $WaitQueue.add(DocumentsWriter.java:1479)
                at
            org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1099)
                at
            org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:777)
                at
            org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:757)
                at
            org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2089)
                at
            org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:204)
                at
            org.elasticsearch.index.shard.service.InternalIndexShard.innerIndex(InternalIndexShard.java:240)
                at
            org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:228)
                at
            org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:125)
                at
            org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:56)
                at
            org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:328)
                at
            org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.access$400(TransportShardReplicationOperationAction.java:198)
                at
            org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:252)
                at java.util.concurrent.ThreadPoolExecutor
            $Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor
            $Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
    
    
    Thanks,
    
    -- 
    Franz Allan Valencia See | Java Software Engineer 
    franz.see@gmail.com
    LinkedIn: http://www.linkedin.com/in/franzsee
    Twitter: http://www.twitter.com/franz_see

--
Franz Allan Valencia See | Java Software Engineer
franz.see@gmail.com
LinkedIn: http://www.linkedin.com/in/franzsee
Twitter: http://www.twitter.com/franz_see
--
Web Announcements Limited is a company registered in England and Wales,
with company number 05608868, with registered address at 10 Arvon Road,
London, N5 1PR.


(system) #4