/_security/api_key timeout

[seism0saurus (Ulrich Viefhaus)]

Hi,
I encounter a problem with the API key management in Kibana.
The page behind Stack Management -> Security -> API keys does not load. After some time it displays "Could not load API keys."
Under technical details is a "proxy error".
I tried to directly call the API endpoint /_security/api_key in the developer console but get this response:

{
  "statusCode": 502,
  "error": "Bad Gateway",
  "message": "Client request timeout"
}

When I search in the Kibana logfiles I only find one mention of api_key:

 grep -e "api_key" /var/log/kibana/kibana.log 

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.1"},"@timestamp":"2024-01-18T05:17:30.983+01:00","message":"Failed to mark Task alerts_invalidate_api_keys \"Alerts-alerts_invalidate_api_keys\" as running: Request timed out","log":{"level":"ERROR","logger":"plugins.taskManager"},"process":{"pid":23200},"trace":{"id":"20fb114172fcef1bf344bc14f8f1dd25"},"transaction":{"id":"092958f692f4f72b"}}

It is some time ago I was on the API key page so I can't tell the version, it broke. Currently we run 8.12.0.

Has anyone an idea, what's going wrong?

[seism0saurus (Ulrich Viefhaus)]

One more piece of information. If I try to make a direct call to the API with curl and not through the developer console I get an empty reply:

curl -u "secret_user:secret_password" https://my_elastic_server:5601/_security/api_key
curl: (52) Empty reply from server

[seism0saurus (Ulrich Viefhaus)]

I have increased the timeout for calls from Kibana to Elasticsearch to 120 seconds instead of the default (30).

Now calling GET /_security/api_key in the developer console results in this error:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "RecyclerBytesStreamOutput cannot hold more than 2GB of data"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "RecyclerBytesStreamOutput cannot hold more than 2GB of data",
    "suppressed": [
      {
        "type": "illegal_state_exception",
        "reason": "Failed to close the XContentBuilder",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unclosed object or array found"
        }
      }
    ]
  },
  "status": 400
}

There are many such errors in search engines. I'm trying to figure out, what the cause could be. I only want to load the API keys. That should never generate 2 GB of data.