The reply they gave me is as follows: " The reason for this exception is that you have not provided a exception handler to the ioreactor. If no exception handler is provided then for runtime exception and IOException the reactor stops by default. You need to give an handler and override the handle method to return true in case of a Runtime exception so that the IOReactor continues when there is an Runtimeexception".
Can someone please provide me a sample/guide me to what I need to do precisely?
I think your confusion is that client.close() doesn't simply close a connection, it completely destroys the whole client and in particular it tears down the underlying IOReactor. Calling any methods on the client after calling client.close() is a bug in how you're using the client, and it's right to report that the reactor has been stopped in these circumstances.
This is not correct information by the way, I recommend ignoring it. The client configures the IOReactor with appropriate exception handling.
EDIT/CORRECTION: This isn't quite true, sorry, I misunderstood what the poster meant. The client doesn't configure exception handling on the IOReactor itself. Instead, it is careful not to throw any exceptions that would terminate the IOReactor and to properly handle all the exceptions that it does encounter.
Hi David, we get very often that error in production with no reason. I understand that the connection is closed due to an unknown reason, but is there a way to prevent it or to automatically renew the connection?
I can reproduce the error by calling client.close(), but in prod it happens out of the blue. Connection is closed with no reason and we just keep getting those errors.
The message Request cannot be executed; I/O reactor status: STOPPED does not indicate that a connection was closed, it indicates that you have called client.close() while the client was still in use. If a connection is closed you'll get an appropriate exception, maybe Connection reset or something like that.
OK, so is there a way to automatically re-establish the connection when such a connection error occurs? Or maybe some other recommendation - we are using a single connection at the moment, maybe there is some pool API that we could use instead and if would establish new connection in such case?
Yep, just keep on using the same client, it will automatically open new connections if it needs them. It uses a connection pool by default, no extra configuration is necessary. Just don't call client.close() until you're done with it.
David, that's exactly what we do, we never call client.close() manually, I just mentioned this as a way to reproduce the exception. We do use a single client which once every several weeks somehow disconnects and then our app keeps failing with the mentioned exception until we recreate the client. It happens for multiple customers/installations. So it doesn't work as you described, at least with 7.10 version of the client.
I don't really know what to say here. The message you've shared really indicates that the client has been closed while it's still in use.
About the only thing I can think of is that the client is auto-closable so it can be closed without an explicit call to the close() method, simply exiting a try-with-resources block is enough.
Yeah, that's the point, that we do not close it at all, it happens on all servers once every few weeks, if that would be our code, they it would happen all the time. Somehow from some unknown reason the connection is closed and ES client do not re-establishes it.
Is there any way to troubleshoot this issue? This is really affecting our customers and we do not see any other exception in the logs prior to that IO Reactor STOPPED error. We do not close it manually neither. It has to be some low level issue.
Tracing through the code, I believe that exceptional termination of the IOReactor should always result in an ERROR level log message I/O reactor terminated abnormally. I'm still not seeing how this can happen in the REST client itself, it appears to need an exception handler itself to throw an exception which doesn't seem to be possible, or something to be terribly broken with accept() and friends, but it's hard to be sure that I've checked everything. In any case you're not seeing this message, so that rules out exceptional termination of the reactor.
Another possibility appears to be that something throws an Error (e.g. AssertionError or OutOfMemoryError), which will bubble all the way up to the uncaught exception handler. Are you capturing uncaught errors correctly?
Aside from that, I don't see any great troubleshooting steps today. I opened #75525 since I think the troubleshooting difficulty is itself a bug.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
