Too many open files

Elastic Stack Elasticsearch
1

hello, I'm using elasticsearch with storm, Java TransportClient.

I have total 128 threads across machines which communicate with
elasticsearch cluster.

From time to time, error below occurs

org.elasticsearch.common.netty.channel.ChannelException: Failed to create a
selector.
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:143)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:81)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33)
at
org.elasticsearch.transport.netty.NettyTransport.doStart(NettyTransport.java:254)
at
org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at
org.elasticsearch.transport.TransportService.doStart(TransportService.java:92)
at
org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at
org.elasticsearch.client.transport.TransportClient.(TransportClient.java:189)
at
org.elasticsearch.client.transport.TransportClient.(TransportClient.java:125)
at
com.naver.labs.nelo2.notifier.utils.ElasticSearchUtil.prepareElasticSearch(ElasticSearchUtil.java:30)
at
com.naver.labs.nelo2.notifier.bolt.PercolatorBolt.prepare(PercolatorBolt.java:48)
at
backtype.storm.topology.BasicBoltExecutor.prepare(BasicBoltExecutor.java:43)
at backtype.storm.daemon.executor$fn__5641$fn__5653.invoke(executor.clj:690)
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:429)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: too many open files
at sun.nio.ch.IOUtil.makePipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65)
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at java.nio.channels.Selector.open(Selector.java:227)
at
org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.open(SelectorUtil.java:63)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:341)
... 22 more

code itself is very simple

I get client as follows

    Settings settings =
            ImmutableSettings.settingsBuilder().put("cluster.name",

clusterName).put("client.transport.sniff", "true").build();
List transportAddressList = new
ArrayList();
for (String host : ESHost) {
transportAddressList.add(new InetSocketTransportAddress(host,
ESPort));
}
return new TransportClient(settings)
.addTransportAddresses(transportAddressList.toArray(new
InetSocketTransportAddress[transportAddressList.size()]));

and for each execution, it percolates as follows

    return

client.preparePercolate().setIndices(indexName).setDocumentType(projectName).setPercolateDoc(docBuilder().setDoc(log)).setRouting(projectName).setPercolateFilter(FilterBuilders.termFilter("projects",
projectName).cache(true)).execute().actionGet();

ES cluster consists of 5 machines with almost default setting.

What can be the cause of this problem?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL3_U43SQ2KCGFCQ%3DcisaLGQXxdg_r1mzjrcunreaJOj0Ln-%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

2

This is a fairly common problem and not necessarily specific to Elasticsearch. It is simple to solve. In Linux you can increase the operating system's max file descriptor limit. Other Unix-like operating systems have the same concept. You can find how to do this for your specific Linux distribution from a little googling on "linux max file descriptor".

Cheers.

On Jul 17, 2014, at 9:40 PM, Seungjin Lee sweetest0409@gmail.com wrote:

hello, I'm using elasticsearch with storm, Java TransportClient.

I have total 128 threads across machines which communicate with elasticsearch cluster.

From time to time, error below occurs

org.elasticsearch.common.netty.channel.ChannelException: Failed to create a selector.
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:143)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:81)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33)
at org.elasticsearch.transport.netty.NettyTransport.doStart(NettyTransport.java:254)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:92)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:189)
at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:125)
at com.naver.labs.nelo2.notifier.utils.ElasticSearchUtil.prepareElasticSearch(ElasticSearchUtil.java:30)
at com.naver.labs.nelo2.notifier.bolt.PercolatorBolt.prepare(PercolatorBolt.java:48)
at backtype.storm.topology.BasicBoltExecutor.prepare(BasicBoltExecutor.java:43)
at backtype.storm.daemon.executor$fn__5641$fn__5653.invoke(executor.clj:690)
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:429)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: too many open files
at sun.nio.ch.IOUtil.makePipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at java.nio.channels.Selector.open(Selector.java:227)
at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.open(SelectorUtil.java:63)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:341)
... 22 more

code itself is very simple

I get client as follows

    Settings settings =
            ImmutableSettings.settingsBuilder().put("cluster.name", clusterName).put("client.transport.sniff", "true").build();
    List<InetSocketTransportAddress> transportAddressList = new  ArrayList<InetSocketTransportAddress>();
    for (String host : ESHost) {
        transportAddressList.add(new InetSocketTransportAddress(host, ESPort));
    }
    return new TransportClient(settings)
            .addTransportAddresses(transportAddressList.toArray(new InetSocketTransportAddress[transportAddressList.size()]));

and for each execution, it percolates as follows

    return client.preparePercolate().setIndices(indexName).setDocumentType(projectName).setPercolateDoc(docBuilder().setDoc(log)).setRouting(projectName).setPercolateFilter(FilterBuilders.termFilter("projects", projectName).cache(true)).execute().actionGet();

ES cluster consists of 5 machines with almost default setting.

What can be the cause of this problem?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL3_U43SQ2KCGFCQ%3DcisaLGQXxdg_r1mzjrcunreaJOj0Ln-%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/DDFC2392-63BE-49DA-BDE8-48F25185B444%40elasticsearch.com.
For more options, visit https://groups.google.com/d/optout.