Failed to connect to 127.0.0.1: Cannot assign requested address

Hi everybody,

I am testing elasticsearch since a while and all the sudden today, I am
getting a weird error without having changed anything.

My PHP script indexes ~ 180k documents, and currently stops reliably after
about 40k - 50k documents with the error message mentioned in the subject.

There is nothing in the elasticsearch.log.

On the PHP side I am using curl to connect to the service. I do one request
per document, so it works fine for the first 40k or so docs.

This error does not happen on the same document, but very reliably after
40k - 50k docs, after I deleted and re-created the index and even restartet
the ES service.

Previously I have used "localhost:9200" as the address and changed it to
127.0.0.1, which did not fix the problem.

I am using the ES version 0.19.8 on Debian (stock Debian install with ICU
plugin).

Any ideas?

Thx
Martin

--

Martin Rode wrote:

I am testing elasticsearch since a while and all the sudden today, I am
getting a weird error without having changed anything.

My PHP script indexes ~ 180k documents, and currently stops reliably after
about 40k - 50k documents with the error message mentioned in the subject.

There is nothing in the elasticsearch.log.

On the PHP side I am using curl to connect to the service. I do one request
per document, so it works fine for the first 40k or so docs.

This error does not happen on the same document, but very reliably after
40k - 50k docs, after I deleted and re-created the index and even restartet
the ES service.

Can you give us an example of the error you're seeing?

-Drew

--

Drew, what do you mean?

CURL cannot connect to the elasticsearch server and says:

Could not read from http://127.0.0.1:9200/easydb/bilder/43470 (0): Failed
to connect to 127.0.0.1: Cannot assign requested address
*
*

On Friday, October 12, 2012 12:21:21 AM UTC+2, Drew Raines wrote:

Martin Rode wrote:

I am testing elasticsearch since a while and all the sudden today, I am
getting a weird error without having changed anything.

My PHP script indexes ~ 180k documents, and currently stops reliably
after
about 40k - 50k documents with the error message mentioned in the
subject.

There is nothing in the elasticsearch.log.

On the PHP side I am using curl to connect to the service. I do one
request
per document, so it works fine for the first 40k or so docs.

This error does not happen on the same document, but very reliably after
40k - 50k docs, after I deleted and re-created the index and even
restartet
the ES service.

Can you give us an example of the error you're seeing?

-Drew

--

I am using pyes, instead of PHP/curl... and I was get this error after a
while of heavy indexing:

pyes.exceptions.NoServerAvailable

Google that error instead... I believe it's the same resource limits
issue...

In my case, I added code to simply sleep for 60 seconds
after every 500,000 documents indexed... to tell the cluster relax a bit :slight_smile:

Thanks,
Mohamed.

On Friday, October 12, 2012 9:26:57 AM UTC-4, Martin Rode wrote:

Drew, what do you mean?

CURL cannot connect to the elasticsearch server and says:

Could not read from http://127.0.0.1:9200/easydb/bilder/43470 (0): Failed
to connect to 127.0.0.1: Cannot assign requested address
*
*

On Friday, October 12, 2012 12:21:21 AM UTC+2, Drew Raines wrote:

Martin Rode wrote:

I am testing elasticsearch since a while and all the sudden today, I am
getting a weird error without having changed anything.

My PHP script indexes ~ 180k documents, and currently stops reliably
after
about 40k - 50k documents with the error message mentioned in the
subject.

There is nothing in the elasticsearch.log.

On the PHP side I am using curl to connect to the service. I do one
request
per document, so it works fine for the first 40k or so docs.

This error does not happen on the same document, but very reliably
after
40k - 50k docs, after I deleted and re-created the index and even
restartet
the ES service.

Can you give us an example of the error you're seeing?

-Drew

--

Martin Rode wrote:

Drew, what do you mean?

You had given us a description of a problem with no details, but this
looks helpful:

CURL cannot connect to the elasticsearch server and says:

Could not read from http://127.0.0.1:9200/easydb/bilder/43470 (0): Failed
to connect to 127.0.0.1: Cannot assign requested address

Is the ES JVM dying somehow? This suggests that it's not listening
anymore. And if so, you really don't see anything in the log? Try
setting rootLogger to DEBUG in config/logging.yml to see if anything
appears in the log then.

-Drew

--

By chance does your server have selinux and if so did it get changed to
enabled?

On Thursday, October 11, 2012 11:43:32 AM UTC-4, Martin Rode wrote:

Hi everybody,

I am testing elasticsearch since a while and all the sudden today, I am
getting a weird error without having changed anything.

My PHP script indexes ~ 180k documents, and currently
stops reliably after about 40k - 50k documents with the error message
mentioned in the subject.

There is nothing in the elasticsearch.log.

On the PHP side I am using curl to connect to the service. I do one
request per document, so it works fine for the first 40k or so docs.

This error does not happen on the same document, but very reliably after
40k - 50k docs, after I deleted and re-created the index and even restartet
the ES service.

Previously I have used "localhost:9200" as the address and changed it to
127.0.0.1, which did not fix the problem.

I am using the ES version 0.19.8 on Debian (stock Debian install with ICU
plugin).

Any ideas?

Thx
Martin

--

You did not set the SO_REUSEADDR flag, so after "40k -50k documents", you
have forced your system to ran out of sockets. See
also http://php.net/manual/de/function.socket-set-option.php

Jörg

On Thursday, October 11, 2012 5:43:32 PM UTC+2, Martin Rode wrote:

Hi everybody,

I am testing elasticsearch since a while and all the sudden today, I am
getting a weird error without having changed anything.

My PHP script indexes ~ 180k documents, and currently
stops reliably after about 40k - 50k documents with the error message
mentioned in the subject.

There is nothing in the elasticsearch.log.

On the PHP side I am using curl to connect to the service. I do one
request per document, so it works fine for the first 40k or so docs.

This error does not happen on the same document, but very reliably after
40k - 50k docs, after I deleted and re-created the index and even restartet
the ES service.

Previously I have used "localhost:9200" as the address and changed it to
127.0.0.1, which did not fix the problem.

I am using the ES version 0.19.8 on Debian (stock Debian install with ICU
plugin).

Any ideas?

Thx
Martin

--

I was encountering the same problem using cURL in PHP. I was opening and closing a cURL connection with each request to Elasticsearch, and was making hundreds of thousands of requests from the same PHP process.

$this->Curl = curl_init();
// make requests
curl_close( $this->Curl );

When I rewrote my class to use the same cURL connection instance for every request, this problem went away. ( eg. I never call curl_close() in my script, and use $this->Curl for every request ).

I do not know what is limiting the number of cURL connections I'm able to open and close in a single PHP thread. Perhaps someone else can shine some light on this.

As said, that is very basic to socket programming. Operation systems
keep socket addresses reserved after being closed for some minutes, in
order to be prepared for reuse at reopening (TCP state CLOSE_WAIT).
Therefore, you should use persistent connections (HTTP keepalive or
SO_REUSEADDR), otherwise you will quickly consume all the socket
addresses after some thousands request open/close cycles. Note that
socket addresses are usually 16 bit wide, and for user space, you can
go up to around a number of 40-50k sockets totally.

Jörg

Am 06.02.13 09:15, schrieb tbrianjones:

I do not know what is limiting the number of cURL connections I'm able to
open and close in a single PHP thread. Perhaps someone else can shine some
light on this.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.