When Starting Data Frame, there is an error

Elastic Stack Elasticsearch
1

My Elasticsearch is 7.8. I can create a data frame job. but when I start it, I got the error "{"error":{"root_cause":[{"type":"exception","reason":"Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job"}],"type":"exception","reason":"Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job","caused_by":{"type":"file_not_found_exception","reason":"temp/data_frame_analyzer_slurm_idle_predict_job_log_39789 (No such file or directory)"}},"status":500}". I try to get solution, 3Q.

2

Hi @Muqali_He,

Are you using the create data frame API at all?

{"type":"file_not_found_exception","reason":"temp/data_frame_analyzer_slurm_idle_predict_job_log_39789

It looks like you are receiving a file not found exception for the log. I see that in version 7.8 the create was in tech preview, so I'm wondering if upgrading will help. Are you able to try that?

Let us know!

3

The more details.
My version is 7.8.0.
The function such as search, ingest and so on is normal. except ml.
My script's content:

{

        "analysis": {
                "regression": {
                        "dependent_variable": "idle",
                        "training_percent": 90
                }
        },
        "dest": {
                "index": "slurm_nodes_num_predict"
        },
        "source": {
                "index": "slurm_nodes_num"
        },
        "model_memory_limit": "100mb"
}

the following is yy Request:

  1. I use create data frame API to create a job with my script.

I get the response

{"id":"slurm_idle_predict_job","source":{"index":["slurm_nodes_num"],"query":{"match_all":{}}},"dest":{"index":"slurm_nodes_num_predict","results_field":"ml"},"analysis":{"regression":{"dependent_variable":"idle","prediction_field_name":"idle_prediction","training_percent":90.0,"randomize_seed":-7889984481626460692,"loss_function":"mse"}},"model_memory_limit":"100mb","create_time":1721091098093,"version":"7.8.0","allow_lazy_start":false}
  1. I POST /_ml/data_frame/analytics/slurm_idle_predict_job/_start
    I get response
type or paste code here{"error":{"root_cause":[{"type":"exception","reason":"Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job"}],"type":"exception","reason":"Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job","caused_by":{"type":"file_not_found_exception","reason":"temp/data_frame_analyzer_slurm_idle_predict_job_log_39789 (No such file or directory)"}},"status":500}

The Elasticsearch log

[2024-07-16T09:01:54,825][WARN ][r.suppressed             ] [node-12] path: /_ml/data_frame/analytics/slurm_idle_predict_job/_start, params: {id=slurm_idle_predict_job}
org.elasticsearch.transport.RemoteTransportException: [node-10][25.8.99.10:9300][cluster:admin/xpack/ml/data_frame/analytics/start]
Caused by: org.elasticsearch.ElasticsearchException: Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job
        at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:55) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createNativeProcess(NativeMemoryUsageEstimationProcessFactory.java:105) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createAnalyticsProcess(NativeMemoryUsageEstimationProcessFactory.java:67) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createAnalyticsProcess(NativeMemoryUsageEstimationProcessFactory.java:34) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.MemoryUsageEstimationProcessManager.runJob(MemoryUsageEstimationProcessManager.java:85) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.MemoryUsageEstimationProcessManager.lambda$runJobAsync$0(MemoryUsageEstimationProcessManager.java:48) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) ~[elasticsearch-7.8.0.jar:7.8.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.io.FileNotFoundException: temp/data_frame_analyzer_slurm_idle_predict_job_log_39789 (No such file or directory)
        at java.io.FileInputStream.open0(Native Method) ~[?:?]
        at java.io.FileInputStream.open(FileInputStream.java:212) ~[?:?]
        at java.io.FileInputStream.<init>(FileInputStream.java:154) ~[?:?]
        at java.io.FileInputStream.<init>(FileInputStream.java:109) ~[?:?]
        at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:288) ~[?:?]
        at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:277) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:130) ~[?:?]
        at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:97) ~[?:?]
        at org.elasticsearch.xpack.ml.process.ProcessPipes.connectStreams(ProcessPipes.java:132) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createNativeProcess(NativeMemoryUsageEstimationProcessFactory.java:101) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createAnalyticsProcess(NativeMemoryUsageEstimationProcessFactory.java:67) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createAnalyticsProcess(NativeMemoryUsageEstimationProcessFactory.java:34) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.MemoryUsageEstimationProcessManager.runJob(MemoryUsageEstimationProcessManager.java:85) ~[?:?]
        at org.elasticsearch.xpack.ml.dataframe.process.MemoryUsageEstimationProcessManager.lambda$runJobAsync$0(MemoryUsageEstimationProcessManager.java:48) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) ~[elasticsearch-7.8.0.jar:7.8.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) ~[?:?]
4

Thanks for the additional detail @Muqali_He. As the feature is in tech preview in version 7.8 and it looks to be a file not found error for a file you don't need to create, I can't rule out that this is a bug. Given that 7.8 is past EOL I would recommend upgrading to a version where the feature is GA. Are you able to upgrade at all?