hi,
we have an ES index that is accessed by a rails server.. things work well
in general. but every few minutes (once/twice an hour), the client gets a
socket connect failure from the ES server. has anyone seen this and if yes,
what is the solution? our ES host is running healthy with heavy write/read
operations going in parallel from java clients. the only difference i see
is that in the java clients, we have a pool of Client objects that we reuse
and dont create a new client for each operation.
are there any best practices here? should there be a http connection
pooling between rails client and ES server? it is not trivial to do in
rails as each web request goes to a separate process.
First, from Java there isn't a need to pool Client objects, you can just use a single Client in your application (its perfectly thread safe).
Regarding the rails client, can you possibly monitor the number of open sockets you have on the machine, maybe you are opening and closing sockets from rails where the OS is starting to throttle it? Any chance of trying and use persistent connections from rails?
hi,
we have an ES index that is accessed by a rails server.. things work well in general. but every few minutes (once/twice an hour), the client gets a socket connect failure from the ES server. has anyone seen this and if yes, what is the solution? our ES host is running healthy with heavy write/read operations going in parallel from java clients. the only difference i see is that in the java clients, we have a pool of Client objects that we reuse and dont create a new client for each operation.
are there any best practices here? should there be a http connection pooling between rails client and ES server? it is not trivial to do in rails as each web request goes to a separate process.
thanks for the tip on java client object. i didnt know it was thread safe.
ill take a look at open sockets as you suggested. do you know of any rails
library/gem that lets you do persistent connections to ES? do people
implement their own wrapper/abstraction here?
thanks
On Tue, Aug 7, 2012 at 2:36 PM, Shay Banon kimchy@gmail.com wrote:
First, from Java there isn't a need to pool Client objects, you can just
use a single Client in your application (its perfectly thread safe).
Regarding the rails client, can you possibly monitor the number of open
sockets you have on the machine, maybe you are opening and closing sockets
from rails where the OS is starting to throttle it? Any chance of trying
and use persistent connections from rails?
hi,
we have an ES index that is accessed by a rails server.. things work well
in general. but every few minutes (once/twice an hour), the client gets a
socket connect failure from the ES server. has anyone seen this and if yes,
what is the solution? our ES host is running healthy with heavy write/read
operations going in parallel from java clients. the only difference i see
is that in the java clients, we have a pool of Client objects that we reuse
and dont create a new client for each operation.
are there any best practices here? should there be a http connection
pooling between rails client and ES server? it is not trivial to do in
rails as each web request goes to a separate process.
We've been used rubberband ruby gem for more than 4 months we didn't get
any issue with timeout.
Rubberband gets nodes list from http://node:9200/_nodes before each search
or index in the same request, it also has built-in retry, failover and
others features, maybe this implementation fits better in your Rails env.
thanks for the tip on java client object. i didnt know it was thread safe.
ill take a look at open sockets as you suggested. do you know of any rails
library/gem that lets you do persistent connections to ES? do people
implement their own wrapper/abstraction here?
thanks
On Tue, Aug 7, 2012 at 2:36 PM, Shay Banon kimchy@gmail.com wrote:
First, from Java there isn't a need to pool Client objects, you can just
use a single Client in your application (its perfectly thread safe).
Regarding the rails client, can you possibly monitor the number of open
sockets you have on the machine, maybe you are opening and closing sockets
from rails where the OS is starting to throttle it? Any chance of trying
and use persistent connections from rails?
hi,
we have an ES index that is accessed by a rails server.. things work well
in general. but every few minutes (once/twice an hour), the client gets a
socket connect failure from the ES server. has anyone seen this and if yes,
what is the solution? our ES host is running healthy with heavy write/read
operations going in parallel from java clients. the only difference i see
is that in the java clients, we have a pool of Client objects that we reuse
and dont create a new client for each operation.
are there any best practices here? should there be a http connection
pooling between rails client and ES server? it is not trivial to do in
rails as each web request goes to a separate process.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.