Error 500/502 download the .elser_model_2_linux-x86_64

[Cristian_Pereyra]

• Arch
  • eck 8.15.2 (now 8.16.1 and same issue)
  • operator: 2.11
  • licence: enterprise

Hello everyone. I am trying to download the .elser_model_2_linux-x86_64 model from Kibana. I have tried the following two approaches:

1. Using the download option from Kibana > Machine Learning > Trained Models, I get:

{"statusCode": 500,"error": "Internal Server Error","message": "Request timed out"}

2. Following the reference ELSER – Elastic Learned Sparse EncodeR | Machine Learning in the Elastic Stack [8.15] | Elastic, I attempted:

PUT _inference/sparse_embedding/my-elser-model
 {
 "service": "elser",
  "service_settings": {
 "num_allocations": 1,
     "num_threads": 1
 }
 }

and

PUT _ml/trained_models/.elser_model_2
{
  "input": {
	"field_names": ["message"]
  }
}

But I receive this response:

{"statusCode": 502,"error": "Bad Gateway","message": "Client request timeout"}

From this second approach, the Kibana logs show the following:

[2024-11-26T23:41:45.573+00:00][ERROR][plugins.console] Error: Client request timeout
at Timeout._onTimeout (/usr/share/kibana/node_modules/@kbn/console-plugin/server/lib/proxy_request.js:91:37)
at listOnTimeout (node:internal/timers:573:17)
at processTimers (node:internal/timers:514:7)
[2024-11-26T23:41:45.574+00:00][WARN ][plugins.console] Could not connect to any configured ES node [https://eck-observabilidad-test-es-http.eck8-ob-test.svc:9200]

On the Elasticsearch side, I don't see any logs. What I understand is that Kibana is not managing to connect to Elasticsearch, which is strange because I can see everything in Elasticsearch from Kibana.My questions are:

1. Is any additional configuration required for Kibana to connect to Elasticsearch when I want to download ELSER?

Extra detail:
From the Kibana pod, using curl to https://eck-observabilidad-test-es-http.eck8-ob-test.svc:9200, I connect without any issues, and it returns the Elasticsearch JSON response.

[stephenb (Stephen Brown)]

Hi @Cristian_Pereyra

1. Do you have any machine learning nodes

GET _cat/nodes?v

2. Machine Learning is licensed feature have you enabled a trial license

3. do you have a FW in place...
   Have you tried to download the model artifacts manually from you network/ ECK pod to see if you have connectivity?

See here

ELSER – Elastic Learned Sparse EncodeR | Machine Learning in the Elastic...

[Cristian_Pereyra]

@stephenb Thank you for responding.

the answers are:

1. Yes, I have 3 nodes with all roles assigned, including the ML role.

/_cluster/settings

2. I have an Enterprise license.

3. I'll answer separately:

3.1 Initially, the proxy didn’t allow me to connect from the pod. However, after changing the proxy settings, I was able to download files. For example, I successfully downloaded the metadata file here without any issues (from the pod, to clarify).

3.2 With 3.1 resolved, I decided to download the model using the file-based access method. Currently, the model files are stored on my pod (on all three nodes) at the path ${path.home}/config/models/.

Following those instructions, I have done the following:

1. Download the model artifact files.
2. Put the files into a models subdirectory inside the config directory of your Elasticsearch deployment.
3. Point your Elasticsearch deployment to the model directory by adding the following line to the config/elasticsearch.yml file:

"persistent": {
        "cluster.max_shards_per_node": "3000",
        "xpack.ml.model_repository": "file://${path.home}/config/models/",
        "xpack.monitoring.collection.enabled": "true"
    },

4. Repeat step 2 and step 3 on all master-eligible nodes.
5. Restart the master-eligible nodes one by one.
6. Navigate to the Trained Models page from the main menu or use the global search field in Kibana. ELSER can be found in the list of trained models.
7. Click the "Add trained model" button, select the ELSER model version you downloaded in step 1 and want to deploy, and click "Download." The selected model will be downloaded from the model directory where you placed it in step 2.

( Points 4 and 5 were not necessary since they are already covered by the deployment of the StatefulSet.)

This is the issue I’m currently facing.

In point 7, I am encountering problems. Kibana tries to connect to Elasticsearch but fails, resulting in a 502 error, which I can find in the Kibana logs. I assume this is a Kibana-related issue because the Elasticsearch logs don’t mention anything about it.

This is the configuration that Kibana has, provided by the ECK operator.

message error from kibana

[stephenb (Stephen Brown)]

My suggestion is since you have a valid Enterprise license is to open a support ticket. They'd probably be able to diagnose this fairly quick. I don't see anything obvious at this moment.

[Cristian_Pereyra]

Yes, I thought the same. I wanted to handle them first based on the information available.

Thank you for your time, and I'll let you know how it went.

[stephenb (Stephen Brown)]

Ohhh check the permissions/ owners of the downloaded files Make sure they are readable by the elasticsearch user.

Oh I saw above but that directory is owned by root Perhaps the directory should be owned by elasticsearch as well