Hi,
I have got a map/reduce job which does queries against the elasticsearch
index, and could force some updates on the index. It runs for a while, up
to a certain point (about 2000 documents, and I have tried different
documents), and afterwards I get java.net.BindException. It seems I can not
connect to the the cluster anymore? Any ideas why this may happen? As I
realized there is no connection limit on elasticsearch.
Thanks.
Ilija
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:525)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:102)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:559)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:511)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:489)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:159)
at org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:203)
at com.mendeley.datamining.content.catalogue.AddDocumentsToIndex.<init>(AddDocumentsToIndex.java:24)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.addToIndex(IncrementalDocumentProcess.java:36)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.initilizeSourceDocument(IncrementalDocumentProcess.java:194)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:104)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:320)
at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.mapred.Child.main(Child.java:217)
Is it possible that you are creating a lot of TransportClients and are not
closing them when you are finished?
On Thursday, November 15, 2012 8:20:42 AM UTC-5, Ilija Subasic wrote:
Hi,
I have got a map/reduce job which does queries against the elasticsearch
index, and could force some updates on the index. It runs for a while, up
to a certain point (about 2000 documents, and I have tried different
documents), and afterwards I get java.net.BindException. It seems I can
not connect to the the cluster anymore? Any ideas why this may happen? As I
realized there is no connection limit on elasticsearch.
Thanks.
Ilija
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:525)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:102)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:559)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:511)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:489)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:159)
at org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:203)
at com.mendeley.datamining.content.catalogue.AddDocumentsToIndex.(AddDocumentsToIndex.java:24)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.addToIndex(IncrementalDocumentProcess.java:36)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.initilizeSourceDocument(IncrementalDocumentProcess.java:194)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:104)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:320)
at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.mapred.Child.main(Child.java:217)
Hi,
I am creating a lot of them, but closing them all. Is there a limit on
the client number?
On Thursday, November 15, 2012 9:29:04 PM UTC, Igor Motov wrote:
Is it possible that you are creating a lot of TransportClients and are not
closing them when you are finished?
On Thursday, November 15, 2012 8:20:42 AM UTC-5, Ilija Subasic wrote:
Hi,
I have got a map/reduce job which does queries against the
elasticsearch index, and could force some updates on the index. It runs for
a while, up to a certain point (about 2000 documents, and I have tried
different documents), and afterwards I get java.net.BindException. It
seems I can not connect to the the cluster anymore? Any ideas why this may
happen? As I realized there is no connection limit on elasticsearch.
Thanks.
Ilija
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:525)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:102)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:559)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:511)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:489)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:159)
at org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:203)
at com.mendeley.datamining.content.catalogue.AddDocumentsToIndex.(AddDocumentsToIndex.java:24)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.addToIndex(IncrementalDocumentProcess.java:36)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.initilizeSourceDocument(IncrementalDocumentProcess.java:194)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:104)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:320)
at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.mapred.Child.main(Child.java:217)
you should create only a singleton instance of a TransportClient per JVM.
Note that a TransportClient comes with a complete connection pool (Netty
workers) and a number of thread pools for various ES operations to handle
concurrent accesses. Each TransportClient startup consumes around 100
sockets. Starting and stopping a lot of TransportClients in a short period
of time may sooner or later pile up to the process file descriptors/socket
resource limit, because the OS delays the effective release of closed
sockets for a few minutes. You could watch out for orphaned sockets in
CLOSE_WAIT state if you enter such situations.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.