ML Job Failed: Error opening config file

Elastic Stack Elasticsearch
1

After I submit a straightforward Single Metric Job, it fails to run. No details in the UI. Elasticsearch and Kibana run on seperate hosts, both Windows Server 2016. The main error from the elasticsearch log after submitting the job:
"Error opening config file ${ES_TMPDIR}\limitconfig8601117692750466127.conf"

Here's the full log:
[2018-07-02T11:55:39,703][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [STUDIE03] [logonpred_job3] Loading model snapshot [N/A], job latest_record_timestamp [N/A]
[2018-07-02T11:55:40,377][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [STUDIE03] Successfully set job state to [opened] for job [logonpred_job3]
[2018-07-02T11:55:40,424][INFO ][o.e.x.m.a.TransportPutDatafeedAction] [STUDIE03] Created datafeed [datafeed-logonpred_job3]
[2018-07-02T11:55:40,471][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [logonpred_job3] [autodetect/5100] [CResourceMonitor.cc@104] Setting model memory limit to 4096 MB
[2018-07-02T11:55:40,471][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [logonpred_job3] [autodetect/5100] [CLimits.cc@60] Error opening config file ${ES_TMPDIR}\limitconfig8601117692750466127.conf
[2018-07-02T11:55:40,471][FATAL][o.e.x.m.j.p.l.CppLogMessageHandler] [logonpred_job3] [autodetect/5100] [Main.cc@192] Ml limit config file '${ES_TMPDIR}\limitconfig8601117692750466127.conf' could not be loaded
[2018-07-02T11:55:40,471][INFO ][o.e.x.m.j.p.a.NativeAutodetectProcess] [logonpred_job3] State output finished
[2018-07-02T11:55:40,471][ERROR][o.e.x.m.j.p.a.NativeAutodetectProcess] [logonpred_job3] autodetect process stopped unexpectedly: Error opening config file ${ES_TMPDIR}\limitconfig8601117692750466127.conf
Ml limit config file '${ES_TMPDIR}\limitconfig8601117692750466127.conf' could not be loaded
[2018-07-02T11:55:40,471][WARN ][o.e.x.m.j.p.a.o.AutoDetectResultProcessor] [logonpred_job3] some results not processed due to the termination of autodetect
[2018-07-02T11:55:40,549][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [STUDIE03] Successfully set job state to [failed] for job [logonpred_job3]
[2018-07-02T11:55:40,580][INFO ][o.e.x.m.d.DatafeedManager] Datafeed [datafeed-logonpred_job3] is waiting for job [logonpred_job3] to be opened

2

Additional environment details:

ELK stack 6.2.4

java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

jvm.options contains -Djava.io.tmpdir=${ES_TMPDIR}

ES_TMPDIR path variable is set to specific folder (nothing is created in that folder though).
Elasticsearch runs as Windows service using Local System Account

3

I've changed -Djava.io.tmpdir to point directly to the intended folder. Restarted Elasticsearch and now the ML jobs run.

4

Did you originally install the Elasticsearch service in an older version of Elasticsearch? If you did then I suspect that problem is that versions before 6.2 did not substitute ES_TMPDIR.

The solution I was going to suggest was to remove the service and then reinstall it using the elasticsearch-service.bat from 6.2.4.

However, editing java.io.tmpdir in the registry settings for Elasticsearch would have the same effect as removing and reinstalling the service, so you've found an alternative solution that's just as good.