Rails client getting socket connect timeouts regularly


(T Vinod Gupta) #1

hi,
we have an ES index that is accessed by a rails server.. things work well
in general. but every few minutes (once/twice an hour), the client gets a
socket connect failure from the ES server. has anyone seen this and if yes,
what is the solution? our ES host is running healthy with heavy write/read
operations going in parallel from java clients. the only difference i see
is that in the java clients, we have a pool of Client objects that we reuse
and dont create a new client for each operation.

are there any best practices here? should there be a http connection
pooling between rails client and ES server? it is not trivial to do in
rails as each web request goes to a separate process.

here is the stack trace every time it happens -

/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in
initialize' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:inopen'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in
open_socket' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:227:inconnect'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:294:in
socket' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:159:inrequest'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/adapter/excon.rb:15:in
call' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/response.rb:8:incall'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/request/json.rb:32:in
call' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:210:inrun_request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:93:in
get' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:54:inget'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:79:in
`mget'

thanks


(Shay Banon) #2

First, from Java there isn't a need to pool Client objects, you can just use a single Client in your application (its perfectly thread safe).

Regarding the rails client, can you possibly monitor the number of open sockets you have on the machine, maybe you are opening and closing sockets from rails where the OS is starting to throttle it? Any chance of trying and use persistent connections from rails?

On Aug 6, 2012, at 5:56 PM, T Vinod Gupta tvinod@readypulse.com wrote:

hi,
we have an ES index that is accessed by a rails server.. things work well in general. but every few minutes (once/twice an hour), the client gets a socket connect failure from the ES server. has anyone seen this and if yes, what is the solution? our ES host is running healthy with heavy write/read operations going in parallel from java clients. the only difference i see is that in the java clients, we have a pool of Client objects that we reuse and dont create a new client for each operation.

are there any best practices here? should there be a http connection pooling between rails client and ES server? it is not trivial to do in rails as each web request goes to a separate process.

here is the stack trace every time it happens -

/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in initialize' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:inopen'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in open_socket' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:227:inconnect'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:294:in socket' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:159:inrequest'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/adapter/excon.rb:15:in call' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/response.rb:8:incall'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/request/json.rb:32:in call' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:210:inrun_request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:93:in get' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:54:inget'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:79:in `mget'

thanks


(T Vinod Gupta) #3

thanks for the tip on java client object. i didnt know it was thread safe.

ill take a look at open sockets as you suggested. do you know of any rails
library/gem that lets you do persistent connections to ES? do people
implement their own wrapper/abstraction here?

thanks

On Tue, Aug 7, 2012 at 2:36 PM, Shay Banon kimchy@gmail.com wrote:

First, from Java there isn't a need to pool Client objects, you can just
use a single Client in your application (its perfectly thread safe).

Regarding the rails client, can you possibly monitor the number of open
sockets you have on the machine, maybe you are opening and closing sockets
from rails where the OS is starting to throttle it? Any chance of trying
and use persistent connections from rails?

On Aug 6, 2012, at 5:56 PM, T Vinod Gupta tvinod@readypulse.com wrote:

hi,
we have an ES index that is accessed by a rails server.. things work well
in general. but every few minutes (once/twice an hour), the client gets a
socket connect failure from the ES server. has anyone seen this and if yes,
what is the solution? our ES host is running healthy with heavy write/read
operations going in parallel from java clients. the only difference i see
is that in the java clients, we have a pool of Client objects that we reuse
and dont create a new client for each operation.

are there any best practices here? should there be a http connection
pooling between rails client and ES server? it is not trivial to do in
rails as each web request goes to a separate process.

here is the stack trace every time it happens -

/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in
initialize' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:inopen'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in
open_socket' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:227:inconnect'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:294:in
socket' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:159:inrequest'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/adapter/excon.rb:15:in
call' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/response.rb:8:incall'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/request/json.rb:32:in
call' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:210:inrun_request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:93:in
get' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:54:inget'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:79:in
`mget'

thanks


(Shairon Toledo) #4

We've been used rubberband ruby gem for more than 4 months we didn't get
any issue with timeout.
Rubberband gets nodes list from http://node:9200/_nodes before each search
or index in the same request, it also has built-in retry, failover and
others features, maybe this implementation fits better in your Rails env.

On Tue, Aug 7, 2012 at 6:42 PM, T Vinod Gupta tvinod@readypulse.com wrote:

thanks for the tip on java client object. i didnt know it was thread safe.

ill take a look at open sockets as you suggested. do you know of any rails
library/gem that lets you do persistent connections to ES? do people
implement their own wrapper/abstraction here?

thanks

On Tue, Aug 7, 2012 at 2:36 PM, Shay Banon kimchy@gmail.com wrote:

First, from Java there isn't a need to pool Client objects, you can just
use a single Client in your application (its perfectly thread safe).

Regarding the rails client, can you possibly monitor the number of open
sockets you have on the machine, maybe you are opening and closing sockets
from rails where the OS is starting to throttle it? Any chance of trying
and use persistent connections from rails?

On Aug 6, 2012, at 5:56 PM, T Vinod Gupta tvinod@readypulse.com wrote:

hi,
we have an ES index that is accessed by a rails server.. things work well
in general. but every few minutes (once/twice an hour), the client gets a
socket connect failure from the ES server. has anyone seen this and if yes,
what is the solution? our ES host is running healthy with heavy write/read
operations going in parallel from java clients. the only difference i see
is that in the java clients, we have a pool of Client objects that we reuse
and dont create a new client for each operation.

are there any best practices here? should there be a http connection
pooling between rails client and ES server? it is not trivial to do in
rails as each web request goes to a separate process.

here is the stack trace every time it happens -

/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in
initialize' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:inopen'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:288:in
open_socket' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:227:inconnect'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:294:in
socket' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/excon-0.6.6/lib/excon/connection.rb:159:inrequest'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/adapter/excon.rb:15:in
call' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/response.rb:8:incall'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/request/json.rb:32:in
call' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:210:inrun_request'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/faraday-0.7.6/lib/faraday/connection.rb:93:in
get' /usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:54:inget'
/usr/local/rvm/gems/ruby-1.9.3-p0/gems/elasticsearch-client-0.0.6/lib/elasticsearch.rb:79:in
`mget'

thanks

--

Shairon Toledo
http://hashcode.me


(system) #5