Hi, I'm writing a plugin that implements a ScoreFunction that needs to look
up some data from a separate index. It does that by having a Client
instance injected. This works perfectly in my box, but when I deploy it to
an EC2 cluster, one of the nodes simply hangs when calling the Client.
The output for /_cat/thread_pool is:
those 3 active requests never finish, and even worse, blocks the node
entirely, it stops responding to all other search requests (which get
queued up and eventually the queue fills up and starts rejecting requests).
There is no CPU usage on that hanging node.
Obviously all the nodes are configured identically (deployed through
opsworks).
Any ideas? I guess injecting Client is not the way to go here? Any
alternatives worth trying?
I got this working by creating my own TransportClient instance instead of
using the injected Client.
Still, it would be nice to understand what's going on here, also locking up
the node like this seems like a pretty serious bug.
On Tuesday, February 10, 2015 at 12:27:07 PM UTC, Mauricio Scheffer wrote:
Hi, I'm writing a plugin that implements a ScoreFunction that needs to
look up some data from a separate index. It does that by having a Client
instance injected. This works perfectly in my box, but when I deploy it to
an EC2 cluster, one of the nodes simply hangs when calling the Client.
The output for /_cat/thread_pool is:
those 3 active requests never finish, and even worse, blocks the node
entirely, it stops responding to all other search requests (which get
queued up and eventually the queue fills up and starts rejecting requests).
There is no CPU usage on that hanging node.
Obviously all the nodes are configured identically (deployed through
opsworks).
Any ideas? I guess injecting Client is not the way to go here? Any
alternatives worth trying?
I got this working by creating my own TransportClient instance instead of
using the injected Client.
Still, it would be nice to understand what's going on here, also locking
up the node like this seems like a pretty serious bug.
On Tuesday, February 10, 2015 at 12:27:07 PM UTC, Mauricio Scheffer wrote:
Hi, I'm writing a plugin that implements a ScoreFunction that needs to
look up some data from a separate index. It does that by having a Client
instance injected. This works perfectly in my box, but when I deploy it to
an EC2 cluster, one of the nodes simply hangs when calling the Client.
The output for /_cat/thread_pool is:
those 3 active requests never finish, and even worse, blocks the node
entirely, it stops responding to all other search requests (which get
queued up and eventually the queue fills up and starts rejecting requests).
There is no CPU usage on that hanging node.
Obviously all the nodes are configured identically (deployed through
opsworks).
Any ideas? I guess injecting Client is not the way to go here? Any
alternatives worth trying?
I can't publish this code, but I'll see if I find some time to create a
repro.
Basically the case is as simple as I described before: a ScoreFunction that
uses an injected Client instance to query a separate index.
I got this working by creating my own TransportClient instance instead of
using the injected Client.
Still, it would be nice to understand what's going on here, also locking
up the node like this seems like a pretty serious bug.
On Tuesday, February 10, 2015 at 12:27:07 PM UTC, Mauricio Scheffer wrote:
Hi, I'm writing a plugin that implements a ScoreFunction that needs to
look up some data from a separate index. It does that by having a Client
instance injected. This works perfectly in my box, but when I deploy it to
an EC2 cluster, one of the nodes simply hangs when calling the Client.
The output for /_cat/thread_pool is:
those 3 active requests never finish, and even worse, blocks the node
entirely, it stops responding to all other search requests (which get
queued up and eventually the queue fills up and starts rejecting requests).
There is no CPU usage on that hanging node.
Obviously all the nodes are configured identically (deployed through
opsworks).
Any ideas? I guess injecting Client is not the way to go here? Any
alternatives worth trying?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.