Incremental updates and searches java.net.BindException


(Ilija Subasic) #1

Hi,
I have got a map/reduce job which does queries against the elasticsearch
index, and could force some updates on the index. It runs for a while, up
to a certain point (about 2000 documents, and I have tried different
documents), and afterwards I get java.net.BindException. It seems I can not
connect to the the cluster anymore? Any ideas why this may happen? As I
realized there is no connection limit on elasticsearch.
Thanks.
Ilija

Stacktrace:

java.net.BindException: Cannot assign requested address

at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:525)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:102)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:559)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:511)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:489)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:159)
at org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:203)
at com.mendeley.datamining.content.catalogue.AddDocumentsToIndex.<init>(AddDocumentsToIndex.java:24)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.addToIndex(IncrementalDocumentProcess.java:36)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.initilizeSourceDocument(IncrementalDocumentProcess.java:194)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:104)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:320)
at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.mapred.Child.main(Child.java:217)

--


(Igor Motov) #2

Is it possible that you are creating a lot of TransportClients and are not
closing them when you are finished?

On Thursday, November 15, 2012 8:20:42 AM UTC-5, Ilija Subasic wrote:

Hi,
I have got a map/reduce job which does queries against the elasticsearch
index, and could force some updates on the index. It runs for a while, up
to a certain point (about 2000 documents, and I have tried different
documents), and afterwards I get java.net.BindException. It seems I can
not connect to the the cluster anymore? Any ideas why this may happen? As I
realized there is no connection limit on elasticsearch.
Thanks.
Ilija

Stacktrace:

java.net.BindException: Cannot assign requested address

at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:525)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:102)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:559)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:511)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:489)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:159)
at org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:203)
at com.mendeley.datamining.content.catalogue.AddDocumentsToIndex.(AddDocumentsToIndex.java:24)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.addToIndex(IncrementalDocumentProcess.java:36)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.initilizeSourceDocument(IncrementalDocumentProcess.java:194)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:104)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:320)
at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.mapred.Child.main(Child.java:217)

--


(Ilija Subasic) #3

Hi,
I am creating a lot of them, but closing them all. Is there a limit on
the client number?

On Thursday, November 15, 2012 9:29:04 PM UTC, Igor Motov wrote:

Is it possible that you are creating a lot of TransportClients and are not
closing them when you are finished?

On Thursday, November 15, 2012 8:20:42 AM UTC-5, Ilija Subasic wrote:

Hi,
I have got a map/reduce job which does queries against the
elasticsearch index, and could force some updates on the index. It runs for
a while, up to a certain point (about 2000 documents, and I have tried
different documents), and afterwards I get java.net.BindException. It
seems I can not connect to the the cluster anymore? Any ideas why this may
happen? As I realized there is no connection limit on elasticsearch.
Thanks.
Ilija

Stacktrace:

java.net.BindException: Cannot assign requested address

at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:525)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:139)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:102)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
at org.elasticsearch.common.netty.channel.Channels.connect(Channels.java:642)
at org.elasticsearch.common.netty.channel.AbstractChannel.connect(AbstractChannel.java:204)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:230)
at org.elasticsearch.common.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:183)
at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:559)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:511)
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:489)
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:128)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:305)
at org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:159)
at org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:203)
at com.mendeley.datamining.content.catalogue.AddDocumentsToIndex.(AddDocumentsToIndex.java:24)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.addToIndex(IncrementalDocumentProcess.java:36)
at com.mendeley.datamining.content.catalogue.IncrementalDocumentProcess.initilizeSourceDocument(IncrementalDocumentProcess.java:194)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:104)
at com.mendeley.datamining.content.catalogue.increment.NonExtendedIncrementalCatalogueCreationJob$TitleJournalMapper.map(NonExtendedIncrementalCatalogueCreationJob.java:37)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:320)
at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.mapred.Child.main(Child.java:217)

--


(Jörg Prante) #4

Hi Ilija,

you should create only a singleton instance of a TransportClient per JVM.

Note that a TransportClient comes with a complete connection pool (Netty
workers) and a number of thread pools for various ES operations to handle
concurrent accesses. Each TransportClient startup consumes around 100
sockets. Starting and stopping a lot of TransportClients in a short period
of time may sooner or later pile up to the process file descriptors/socket
resource limit, because the OS delays the effective release of closed
sockets for a few minutes. You could watch out for orphaned sockets in
CLOSE_WAIT state if you enter such situations.

Jörg

--


#5

I am new to ES2.0 I got the exact same error as above. And, I haven't even gotten it running yet. Any help will be appreciated.


(system) #6