Elastic OpenAI Connector Local LLM

Elastic Stack Elasticsearch
1

Hello everyone,

i have an elastic instance where i try to set up a custom OpenAI connector to a machine with a locally hosted LLM to use within the AI Assistant. Since 8.17 it is possible to use an OpenAI compatible API connector to accomplish this.

Unfortunately, when i send a prompt from the AI Assistant, i get a rather generic errors in the AI Assistant Chat:

ActionsClientChatOpenAI: an error occurred while running the action - Status code: undefined. Message: Unexpected API Error: ERR_CANCELED - canceled.

ActionsClientChatOpenAI: an error occurred while running the action - Unexpected API Error: - Request was aborted.

I can see the incoming request from Kibana in the logs of lm-studio (the application which hosts the llm). After a short period of time however (after the request from Kibana comes in on the llm-host), a log in lm-studio appears which states "Client disconnected. Stopping generation ..." even before the llm is able to finish generating a response. When sending a prompt from the kibana host to the llm via curl directly to the API endpoint, i get a response back, thus i suspect the issue is related to Elastic/Kibana. The test of the connector succeeds.

Thank you in advance!

2 Likes
2

Same problem. Kibana connecting to LLM and receiving responses stream. But Kibana AI assistant chat not showing the text. And after 60s wait - shows "Request was aborted." error. Changes in xpack.actions.responseTimeout parameter is not solve the problem

1 Like
3

Sounds somewhat related. However Kibana AI Assistant Chat does not wait that long in my case. I get the error messages a couple of seconds later after i sent the chat message to the llm server. Kibana still somewhat behaves like it time-outs prematurely and setting parameter xpack.actions.responseTimeout did not yield any change in behavior. I also removed any other complexity already such as the reverse proxy which handles tls. Kibana directly speaks to the API of lm-studio hosting the model.

4

Update: Although i switched from lm-studio to the text-generation-webui (to somewhat try to address the API implications in the error message), Kibana still responds with the initial error message in the AI Assistant after five to ten seconds:

ActionsClientChatOpenAI: an error occurred while running the action - Status code: undefined. Message: Unexpected API Error: ERR_CANCELED - canceled

Kibana prematurely aborts the request (i.e. responds with the error and does not wait for the response) while the LLM is still generating its response on the other host.