# Search in plugin hangs **URL:** \ **Category:** Elasticsearch\ **Created:** [February 10, 2015, 12:27pm UTC](https://discuss.elastic.co/t/search-in-plugin-hangs/22061 "2015-02-10T12:27:06Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Mauricio\_Scheffer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mauricio_scheffer/32/905_2.png) [@Mauricio\_Scheffer](https://discuss.elastic.co/u/Mauricio_Scheffer)\ **Post date:** [February 10, 2015, 12:27pm UTC](https://discuss.elastic.co/t/search-in-plugin-hangs/22061/1 "2015-02-10T12:27:06Z")
Hi, I'm writing a plugin that implements a ScoreFunction that needs to look up some data from a separate index. It does that by having a Client instance injected. This works perfectly in my box, but when I deploy it to an EC2 cluster, one of the nodes simply hangs when calling the Client. The output for /\_cat/thread\_pool is: elasticsearch-cluster3.localdomain 127.0.1.1 0 0 0 0 0 0 3 18 0 elasticsearch-cluster2.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0 elasticsearch-cluster1.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0 those 3 active requests never finish, and even worse, blocks the node entirely, it stops responding to all other search requests (which get queued up and eventually the queue fills up and starts rejecting requests). There is no CPU usage on that hanging node. Obviously all the nodes are configured identically (deployed through opsworks). Any ideas? I guess injecting Client is not the way to go here? Any alternatives worth trying? Thanks, Mauricio -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0350f575-7c4c-4d88-a471-2ff4d8eeb764%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0350f575-7c4c-4d88-a471-2ff4d8eeb764%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Mauricio\_Scheffer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mauricio_scheffer/32/905_2.png) [@Mauricio\_Scheffer](https://discuss.elastic.co/u/Mauricio_Scheffer)\ **Post date:** [February 12, 2015, 11:58am UTC](https://discuss.elastic.co/t/search-in-plugin-hangs/22061/2 "2015-02-12T11:58:38Z")
I got this working by creating my own TransportClient instance instead of using the injected Client. Still, it would be nice to understand what's going on here, also locking up the node like this seems like a pretty serious bug. On Tuesday, February 10, 2015 at 12:27:07 PM UTC, Mauricio Scheffer wrote: > Hi, I'm writing a plugin that implements a ScoreFunction that needs to > look up some data from a separate index. It does that by having a Client > instance injected. This works perfectly in my box, but when I deploy it to > an EC2 cluster, one of the nodes simply hangs when calling the Client. > The output for /\_cat/thread\_pool is: > > elasticsearch-cluster3.localdomain 127.0.1.1 0 0 0 0 0 0 3 18 0 > elasticsearch-cluster2.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0 > elasticsearch-cluster1.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0 > > those 3 active requests never finish, and even worse, blocks the node > entirely, it stops responding to all other search requests (which get > queued up and eventually the queue fills up and starts rejecting requests). > There is no CPU usage on that hanging node. > Obviously all the nodes are configured identically (deployed through > opsworks). > > Any ideas? I guess injecting Client is not the way to go here? Any > alternatives worth trying? > > Thanks, > Mauricio -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 12, 2015, 4:45pm UTC](https://discuss.elastic.co/t/search-in-plugin-hangs/22061/3 "2015-02-12T16:45:10Z")
Can you publish your code so it can be reproduced? Then you might get feedback. Jörg On Thu, Feb 12, 2015 at 12:58 PM, Mauricio Scheffer \< [mauricioscheffer@gmail.com](mailto:mauricioscheffer@gmail.com)\> wrote: > I got this working by creating my own TransportClient instance instead of > using the injected Client. > Still, it would be nice to understand what's going on here, also locking > up the node like this seems like a pretty serious bug. > > On Tuesday, February 10, 2015 at 12:27:07 PM UTC, Mauricio Scheffer wrote: > > > Hi, I'm writing a plugin that implements a ScoreFunction that needs to > > look up some data from a separate index. It does that by having a Client > > instance injected. This works perfectly in my box, but when I deploy it to > > an EC2 cluster, one of the nodes simply hangs when calling the Client. > > The output for /\_cat/thread\_pool is: > > > > elasticsearch-cluster3.localdomain 127.0.1.1 0 0 0 0 0 0 3 18 0 > > elasticsearch-cluster2.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0 > > elasticsearch-cluster1.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0 > > > > those 3 active requests never finish, and even worse, blocks the node > > entirely, it stops responding to all other search requests (which get > > queued up and eventually the queue fills up and starts rejecting requests). > > There is no CPU usage on that hanging node. > > Obviously all the nodes are configured identically (deployed through > > opsworks). > > > > Any ideas? I guess injecting Client is not the way to go here? Any > > alternatives worth trying? > > > > Thanks, > > Mauricio > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWYq%3DC9EoEqCQk2FTQ-i7WVnaTUgQC3LvTmGW14VPnQA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWYq%3DC9EoEqCQk2FTQ-i7WVnaTUgQC3LvTmGW14VPnQA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Mauricio\_Scheffer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mauricio_scheffer/32/905_2.png) [@Mauricio\_Scheffer](https://discuss.elastic.co/u/Mauricio_Scheffer)\ **Post date:** [February 13, 2015, 5:40pm UTC](https://discuss.elastic.co/t/search-in-plugin-hangs/22061/4 "2015-02-13T17:40:20Z")
I can't publish this code, but I'll see if I find some time to create a repro. Basically the case is as simple as I described before: a ScoreFunction that uses an injected Client instance to query a separate index. Cheers -- Mauricio On Thu, Feb 12, 2015 at 4:45 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > Can you publish your code so it can be reproduced? Then you might get > feedback. > > Jörg > > On Thu, Feb 12, 2015 at 12:58 PM, Mauricio Scheffer \< > [mauricioscheffer@gmail.com](mailto:mauricioscheffer@gmail.com)\> wrote: > > > I got this working by creating my own TransportClient instance instead of > > using the injected Client. > > Still, it would be nice to understand what's going on here, also locking > > up the node like this seems like a pretty serious bug. > > > > On Tuesday, February 10, 2015 at 12:27:07 PM UTC, Mauricio Scheffer wrote: > > > > > Hi, I'm writing a plugin that implements a ScoreFunction that needs to > > > look up some data from a separate index. It does that by having a Client > > > instance injected. This works perfectly in my box, but when I deploy it to > > > an EC2 cluster, one of the nodes simply hangs when calling the Client. > > > The output for /\_cat/thread\_pool is: > > > > > > elasticsearch-cluster3.localdomain 127.0.1.1 0 0 0 0 0 0 3 18 0 > > > elasticsearch-cluster2.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0 > > > elasticsearch-cluster1.localdomain 127.0.1.1 0 0 0 0 0 0 0 0 0 > > > > > > those 3 active requests never finish, and even worse, blocks the node > > > entirely, it stops responding to all other search requests (which get > > > queued up and eventually the queue fills up and starts rejecting requests). > > > There is no CPU usage on that hanging node. > > > Obviously all the nodes are configured identically (deployed through > > > opsworks). > > > > > > Any ideas? I guess injecting Client is not the way to go here? Any > > > alternatives worth trying? > > > > > > Thanks, > > > Mauricio > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0a29cd69-bb0d-4a60-9563-28fa694f5a85%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/5QHqc9RNsZ0/unsubscribe](https://groups.google.com/d/topic/elasticsearch/5QHqc9RNsZ0/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWYq%3DC9EoEqCQk2FTQ-i7WVnaTUgQC3LvTmGW14VPnQA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWYq%3DC9EoEqCQk2FTQ-i7WVnaTUgQC3LvTmGW14VPnQA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWYq%3DC9EoEqCQk2FTQ-i7WVnaTUgQC3LvTmGW14VPnQA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWYq%3DC9EoEqCQk2FTQ-i7WVnaTUgQC3LvTmGW14VPnQA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAF0\_A3EYVmAF9QXmW70UJtYXDs4\_F72tfLmaop4msf9TP0UURw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAF0_A3EYVmAF9QXmW70UJtYXDs4_F72tfLmaop4msf9TP0UURw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:32am UTC](https://discuss.elastic.co/t/search-in-plugin-hangs/22061/5 "2017-07-06T00:32:53Z")