System getting stuck using elastic search

Hi all,

    We are getting a strange issue when we use the elastic search

in our systems. Following is the exception which is thrown ..

    I have Singleton  Transport Client instance  which is

instantiated at the server startup . Have set the openfiles setting
to unlimited . But still we are getting the following error.
Application server is not responding because of this issue and we had
to restart all the servers. This issue coming most of the times so we
had to switch off the elastic search for now to carry on normal
process

  Your help is highly appreciated.

Thanks,
Prashanth

WARN org.elasticsearch.transport.netty - [Tzabaoth] Exception caught
on netty layer [[id: 0x31ed2fda]]
org.elasticsearch.common.netty.channel.ChannelException: Failed to
create a selector.
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.register(NioClientSocketPipelineSink.java:206)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:
163)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:
113)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:
574)
at
org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:
541)
at
org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:
210)
at
org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:
227)
at
org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:
188)
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:
550)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
503)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
482)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:
124)
at org.elasticsearch.client.transport.TransportClientNodesService
$SimpleNodeSampler.sample(TransportClientNodesService.java:262)
at org.elasticsearch.client.transport.TransportClientNodesService
$ScheduledNodeSampler.run(TransportClientNodesService.java:245)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:
18)
at java.nio.channels.Selector.open(Selector.java:209)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.register(NioClientSocketPipelineSink.java:204)
... 16 more
2012-04-18 15:42:31,074 [elasticsearch[cached]-pool-1805-thread-1]
(Log4jESLogger.java:105) WARN org.elasticsearch.transport.netty -
[Tzabaoth] Exception caught on netty layer [[id: 0x4ae0aad1]]
org.elasticsearch.common.netty.channel.ChannelException: Failed to
create a selector.
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.register(NioClientSocketPipelineSink.java:206)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:
163)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:
113)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:
574)
at
org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:
541)
at
org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:
210)
at
org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:
227)
at
org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:
188)
at
org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:
550)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
503)
at
org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
482)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:
124)
at org.elasticsearch.client.transport.TransportClientNodesService
$SimpleNodeSampler.sample(TransportClientNodesService.java:262)
at org.elasticsearch.client.transport.TransportClientNodesService
$ScheduledNodeSampler.run(TransportClientNodesService.java:245)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:
18)
at java.nio.channels.Selector.open(Selector.java:209)
at
org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.register(NioClientSocketPipelineSink.java:204)
... 16 more

it seems like the open files limit has not been set correctly...

On Fri, Apr 20, 2012 at 1:07 PM, Prashanth prashanth.byrapu@gmail.comwrote:

Hi all,

   We are getting a strange issue when we use the elastic search

in our systems. Following is the exception which is thrown ..

   I have Singleton  Transport Client instance  which is

instantiated at the server startup . Have set the openfiles setting
to unlimited . But still we are getting the following error.
Application server is not responding because of this issue and we had
to restart all the servers. This issue coming most of the times so we
had to switch off the Elasticsearch for now to carry on normal
process

 Your help is highly appreciated.

Thanks,
Prashanth

WARN org.elasticsearch.transport.netty - [Tzabaoth] Exception caught
on netty layer [[id: 0x31ed2fda]]
org.elasticsearch.common.netty.channel.ChannelException: Failed to
create a selector.
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.register(NioClientSocketPipelineSink.java:206)
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:
163)
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:
113)
at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:
574)
at
org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:
541)
at

org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:
210)
at

org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:
227)
at

org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:
188)
at

org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:
550)
at

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
503)
at

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
482)
at

org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:
124)
at org.elasticsearch.client.transport.TransportClientNodesService
$SimpleNodeSampler.sample(TransportClientNodesService.java:262)
at org.elasticsearch.client.transport.TransportClientNodesService
$ScheduledNodeSampler.run(TransportClientNodesService.java:245)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:
18)
at java.nio.channels.Selector.open(Selector.java:209)
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.register(NioClientSocketPipelineSink.java:204)
... 16 more
2012-04-18 15:42:31,074 [elasticsearch[cached]-pool-1805-thread-1]
(Log4jESLogger.java:105) WARN org.elasticsearch.transport.netty -
[Tzabaoth] Exception caught on netty layer [[id: 0x4ae0aad1]]
org.elasticsearch.common.netty.channel.ChannelException: Failed to
create a selector.
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.register(NioClientSocketPipelineSink.java:206)
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:
163)
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:
113)
at

org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:
574)
at
org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:
541)
at

org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:
210)
at

org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:
227)
at

org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:
188)
at

org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:
550)
at

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
503)
at

org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:
482)
at

org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:
124)
at org.elasticsearch.client.transport.TransportClientNodesService
$SimpleNodeSampler.sample(TransportClientNodesService.java:262)
at org.elasticsearch.client.transport.TransportClientNodesService
$ScheduledNodeSampler.run(TransportClientNodesService.java:245)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
at
sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:
18)
at java.nio.channels.Selector.open(Selector.java:209)
at

org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink
$Boss.register(NioClientSocketPipelineSink.java:204)
... 16 more