# IO Reactor errors closing connection **URL:** \ **Category:** Elasticsearch\ **Created:** [July 20, 2021, 10:02am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148 "2021-07-20T10:02:33Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Mariusz\_Pala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mariusz_pala/32/34663_2.png) [@Mariusz\_Pala](https://discuss.elastic.co/u/Mariusz_Pala)\ **Post date:** [July 20, 2021, 10:02am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/1 "2021-07-20T10:02:33Z")
Hi, I asked ES about the IOReactor connection issues(Request cannot be executed; I/O reactor status: STOPPED) and as per an answer in my ticket [RestHighLevelClient disconnection issues · Issue #75364 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/75364), they asked me to put my question here. The reply they gave me is as follows: " The reason for this exception is that you have not provided a exception handler to the ioreactor. If no exception handler is provided then for runtime exception and IOException the reactor stops by default. You need to give an handler and override the `handle` method to return `true` in case of a Runtime exception so that the IOReactor continues when there is an Runtimeexception". Can someone please provide me a sample/guide me to what I need to do precisely? Thank in advance for any help! Mariusz ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 20, 2021, 10:17am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/2 "2021-07-20T10:17:29Z")
Hi again @Mariusz_Pala 🙂 I think your confusion is that `client.close()` doesn't simply close a connection, it completely destroys the whole client and in particular it tears down the underlying IOReactor. Calling any methods on the client after calling `client.close()` is a bug in how you're using the client, and it's right to report that the reactor has been stopped in these circumstances. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 20, 2021, 10:21am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/3 "2021-07-20T10:21:11Z")
> [@Mariusz\_Pala](#): > > The reason for this exception is that you have not provided a exception handler to the ioreactor. This is not correct information by the way, I recommend ignoring it. The client configures the IOReactor with appropriate exception handling. **EDIT/CORRECTION** : This isn't quite true, sorry, I misunderstood what the poster meant. The client doesn't configure exception handling on the IOReactor itself. Instead, it is careful not to throw any exceptions that would terminate the IOReactor and to properly handle all the exceptions that it does encounter. ---
**Author:** ![Mariusz\_Pala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mariusz_pala/32/34663_2.png) [@Mariusz\_Pala](https://discuss.elastic.co/u/Mariusz_Pala)\ **Post date:** [July 20, 2021, 10:24am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/4 "2021-07-20T10:24:16Z")
Hi David, we get very often that error in production with no reason. I understand that the connection is closed due to an unknown reason, but is there a way to prevent it or to automatically renew the connection? I can reproduce the error by calling client.close(), but in prod it happens out of the blue. Connection is closed with no reason and we just keep getting those errors. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 20, 2021, 10:27am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/5 "2021-07-20T10:27:45Z")
The message `Request cannot be executed; I/O reactor status: STOPPED` does not indicate that a connection was closed, it indicates that you have called `client.close()` while the client was still in use. If a connection is closed you'll get an appropriate exception, maybe `Connection reset` or something like that. ---
**Author:** ![Mariusz\_Pala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mariusz_pala/32/34663_2.png) [@Mariusz\_Pala](https://discuss.elastic.co/u/Mariusz_Pala)\ **Post date:** [July 20, 2021, 10:31am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/6 "2021-07-20T10:31:11Z")
OK, so is there a way to automatically re-establish the connection when such a connection error occurs? Or maybe some other recommendation - we are using a single connection at the moment, maybe there is some pool API that we could use instead and if would establish new connection in such case? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 20, 2021, 10:36am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/7 "2021-07-20T10:36:48Z")
Yep, just keep on using the same client, it will automatically open new connections if it needs them. It uses a connection pool by default, no extra configuration is necessary. Just don't call `client.close()` until you're done with it. ---
**Author:** ![Mariusz\_Pala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mariusz_pala/32/34663_2.png) [@Mariusz\_Pala](https://discuss.elastic.co/u/Mariusz_Pala)\ **Post date:** [July 20, 2021, 10:41am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/8 "2021-07-20T10:41:59Z")
David, that's exactly what we do, we never call `client.close()` manually, I just mentioned this as a way to reproduce the exception. We do use a single client which once every several weeks somehow disconnects and then our app keeps failing with the mentioned exception until we recreate the client. It happens for multiple customers/installations. So it doesn't work as you described, at least with 7.10 version of the client. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 20, 2021, 10:58am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/9 "2021-07-20T10:58:38Z")
I don't really know what to say here. The message you've shared really indicates that the client has been closed while it's still in use. About the only thing I can think of is that the client is auto-closable so it can be closed without an explicit call to the `close()` method, simply exiting a try-with-resources block is enough. ---
**Author:** ![Mariusz\_Pala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mariusz_pala/32/34663_2.png) [@Mariusz\_Pala](https://discuss.elastic.co/u/Mariusz_Pala)\ **Post date:** [July 20, 2021, 11:04am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/10 "2021-07-20T11:04:16Z")
Yeah, that's the point, that we do not close it at all, it happens on all servers once every few weeks, if that would be our code, they it would happen all the time. Somehow from some unknown reason the connection is closed and ES client do not re-establishes it. ---
**Author:** ![Mariusz\_Pala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mariusz_pala/32/34663_2.png) [@Mariusz\_Pala](https://discuss.elastic.co/u/Mariusz_Pala)\ **Post date:** [July 20, 2021, 11:44am UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/11 "2021-07-20T11:44:43Z")
Is there any way to troubleshoot this issue? This is really affecting our customers and we do not see any other exception in the logs prior to that IO Reactor STOPPED error. We do not close it manually neither. It has to be some low level issue. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 20, 2021, 12:18pm UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/12 "2021-07-20T12:18:22Z")
Tracing through the code, I believe that exceptional termination of the IOReactor should [always result](https://github.com/apache/httpasyncclient/blob/4.1.4/httpasyncclient/src/main/java/org/apache/http/impl/nio/client/CloseableHttpAsyncClientBase.java#L66) in an `ERROR` level log message `I/O reactor terminated abnormally`. I'm still not seeing how this can happen in the REST client itself, it appears to need an exception handler itself to throw an exception which doesn't seem to be possible, or something to be terribly broken with `accept()` and friends, but it's hard to be sure that I've checked everything. In any case you're not seeing this message, so that rules out exceptional termination of the reactor. Another possibility appears to be that something throws an `Error` (e.g. `AssertionError` or `OutOfMemoryError`), which will bubble all the way up to the uncaught exception handler. Are you capturing uncaught errors correctly? Aside from that, I don't see any great troubleshooting steps today. I opened [#75525](https://github.com/elastic/elasticsearch/issues/75525) since I think the troubleshooting difficulty is itself a bug. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 17, 2021, 12:19pm UTC](https://discuss.elastic.co/t/io-reactor-errors-closing-connection/279148/13 "2021-08-17T12:19:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.