POST API calls to the endpoint
result in the following 503 Error, no matter the content of the query:
error: "Service Unavailable" message: "Request timed out" statusCode: 503
As soon as we get a 503 Error, if you click the "Save Query" button again, we get the below 409 (Conflict) Error. This tells us that the previous attempt to save our query actually worked, despite returning a 503. If you view your Saved Queries, we can also verify that the first attempt to save worked fine.
error: "Conflict" message: "[query:Example Query]: version conflict, document already exists (current version ): version_conflict_engine_exception" statusCode: 409
Elasticsearch does not show any signs of memory/cpu pressure, it does not have any search or index times in excess of a couple seconds, yet Kibana shows that the Client Response times are regularly shooting up to 30000ms (ie 503 Timeout error), which is how long our Kibana timeout is.
Other Kibana activities such as querying data works totally fine. It is really just POST calls to
/api/saved_objects that are exhibiting this behavior.
I have been unable to debug, and am wondering if there are any known bugs that could explain this. Thoughts I had were maybe the
.kibana index was corrupted in some way, it is related to the version, or our ingress controller to Kibana is resulting in the timeout (although I looked at our performance metrics and it does not appear to be the case).
Edit: This cluster is deployed and managed via the ECK Operator (image