My Elasticsearch is 7.8. I can create a data frame job. but when I start it, I got the error "{"error":{"root_cause":[{"type":"exception","reason":"Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job"}],"type":"exception","reason":"Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job","caused_by":{"type":"file_not_found_exception","reason":"temp/data_frame_analyzer_slurm_idle_predict_job_log_39789 (No such file or directory)"}},"status":500}". I try to get solution, 3Q.
Hi @Muqali_He,
Are you using the create data frame API at all?
{"type":"file_not_found_exception","reason":"temp/data_frame_analyzer_slurm_idle_predict_job_log_39789
It looks like you are receiving a file not found exception for the log. I see that in version 7.8 the create was in tech preview, so I'm wondering if upgrading will help. Are you able to try that?
Let us know!
The more details.
My version is 7.8.0.
The function such as search, ingest and so on is normal. except ml.
My script's content:
{
"analysis": {
"regression": {
"dependent_variable": "idle",
"training_percent": 90
}
},
"dest": {
"index": "slurm_nodes_num_predict"
},
"source": {
"index": "slurm_nodes_num"
},
"model_memory_limit": "100mb"
}
the following is yy Request:
- I use create data frame API to create a job with my script.
I get the response
{"id":"slurm_idle_predict_job","source":{"index":["slurm_nodes_num"],"query":{"match_all":{}}},"dest":{"index":"slurm_nodes_num_predict","results_field":"ml"},"analysis":{"regression":{"dependent_variable":"idle","prediction_field_name":"idle_prediction","training_percent":90.0,"randomize_seed":-7889984481626460692,"loss_function":"mse"}},"model_memory_limit":"100mb","create_time":1721091098093,"version":"7.8.0","allow_lazy_start":false}
- I POST /_ml/data_frame/analytics/slurm_idle_predict_job/_start
I get response
type or paste code here{"error":{"root_cause":[{"type":"exception","reason":"Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job"}],"type":"exception","reason":"Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job","caused_by":{"type":"file_not_found_exception","reason":"temp/data_frame_analyzer_slurm_idle_predict_job_log_39789 (No such file or directory)"}},"status":500}
The Elasticsearch log
[2024-07-16T09:01:54,825][WARN ][r.suppressed ] [node-12] path: /_ml/data_frame/analytics/slurm_idle_predict_job/_start, params: {id=slurm_idle_predict_job}
org.elasticsearch.transport.RemoteTransportException: [node-10][25.8.99.10:9300][cluster:admin/xpack/ml/data_frame/analytics/start]
Caused by: org.elasticsearch.ElasticsearchException: Failed to launch data frame analytics memory usage estimation process for job slurm_idle_predict_job
at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:55) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createNativeProcess(NativeMemoryUsageEstimationProcessFactory.java:105) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createAnalyticsProcess(NativeMemoryUsageEstimationProcessFactory.java:67) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createAnalyticsProcess(NativeMemoryUsageEstimationProcessFactory.java:34) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.MemoryUsageEstimationProcessManager.runJob(MemoryUsageEstimationProcessManager.java:85) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.MemoryUsageEstimationProcessManager.lambda$runJobAsync$0(MemoryUsageEstimationProcessManager.java:48) ~[?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) ~[elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.io.FileNotFoundException: temp/data_frame_analyzer_slurm_idle_predict_job_log_39789 (No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:?]
at java.io.FileInputStream.open(FileInputStream.java:212) ~[?:?]
at java.io.FileInputStream.<init>(FileInputStream.java:154) ~[?:?]
at java.io.FileInputStream.<init>(FileInputStream.java:109) ~[?:?]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:288) ~[?:?]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:277) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:130) ~[?:?]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:97) ~[?:?]
at org.elasticsearch.xpack.ml.process.ProcessPipes.connectStreams(ProcessPipes.java:132) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createNativeProcess(NativeMemoryUsageEstimationProcessFactory.java:101) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createAnalyticsProcess(NativeMemoryUsageEstimationProcessFactory.java:67) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.NativeMemoryUsageEstimationProcessFactory.createAnalyticsProcess(NativeMemoryUsageEstimationProcessFactory.java:34) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.MemoryUsageEstimationProcessManager.runJob(MemoryUsageEstimationProcessManager.java:85) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.process.MemoryUsageEstimationProcessManager.lambda$runJobAsync$0(MemoryUsageEstimationProcessManager.java:48) ~[?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) ~[elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
at java.lang.Thread.run(Thread.java:832) ~[?:?]
Thanks for the additional detail @Muqali_He. As the feature is in tech preview in version 7.8 and it looks to be a file not found error for a file you don't need to create, I can't rule out that this is a bug. Given that 7.8 is past EOL I would recommend upgrading to a version where the feature is GA. Are you able to upgrade at all?