I get an error creating the most basic machine learning job. In kibana it says "could not open job an internal server error occured." In the elastic logs this is what it says. I can launch autodetect plugin from the cli just fine. Has anyone else ever seen this? Running 5.5 for everything.
[2017-08-28T10:29:15,871][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/17563] [CDetachedProcessSpawner.cc@283] Cannot execute './autodetect': No such file or directory
[2017-08-28T10:29:15,872][ERROR][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/17563] [CCommandProcessor.cc@107] Failed to start process './autodetect'
[2017-08-28T10:29:26,214][ERROR][o.e.x.m.j.p.a.NativeAutodetectProcessFactory] Failed to launch autodetect for job http_response_test
[2017-08-28T10:29:26,280][WARN ][o.e.x.p.PersistentTasksNodeService] [test-cld-es01] task job-http_response_test failed with an exception
org.elasticsearch.ElasticsearchException: Failed to launch autodetect for job http_response_test
at org.elasticsearch.xpack.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:47) ~[?:?]
at org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcessFactory.createNativeProcess(NativeAutodetectProcessFactory.java:116) ~[?:?]
at org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcessFactory.createAutodetectProcess(NativeAutodetectProcessFactory.java:76) ~[?:?]
at org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager.create(AutodetectProcessManager.java:333) ~[?:?]
at org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$1.lambda$doRun$0(AutodetectProcessManager.java:277) ~[?:?]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_91]
at org.elasticsearch.xpack.ml.job.process.autodetect.AutodetectProcessManager$1.doRun(AutodetectProcessManager.java:276) ~[?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.1.jar:5.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.io.FileNotFoundException: /tmp/autodetect_http_response_test_log_17488 (No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_91]
at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_91]
at java.io.FileInputStream.(FileInputStream.java:138) ~[?:1.8.0_91]
at java.io.FileInputStream.(FileInputStream.java:93) ~[?:1.8.0_91]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:297) ~[?:?]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:286) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_91]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:139) ~[?:?]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:106) ~[?:?]
at org.elasticsearch.xpack.ml.job.process.ProcessPipes.connectStreams(ProcessPipes.java:140) ~[?:?]
at org.elasticsearch.xpack.ml.job.process.autodetect.NativeAutodetectProcessFactory.createNativeProcess(NativeAutodetectProcessFactory.java:112) ~[?:?]
Under the elasticsearch installation do you have a directory ES_HOME/plugins/x-pack/platform/<your platform>/bin where <your platform> is one of linux-x86_64, windows-x86_64 or darwin-x86_64. Can you list the contents? You should have a program called autodetect it needs to have execute permission.
What is the output from ES_HOME/plugins/x-pack/platform/<your platform>/bin/autodetect --help
Usage: autodetect [options] [+ [by ]]
Options::
--help Display this information and exit
--version Display version information and exit
--limitconfig arg Optional limit config file
--modelconfig arg Optional model config file
--fieldconfig arg Optional field config file
--modelplotconfig arg Optional model plot config file
--jobid arg ID of the job this process is associated with
--logProperties arg Optional logger properties file
Your original log message indicated that the autodetect program could not be found so good to know that it's present.
Does the user elasticsearch runs as have permission to execute a program in that directory, can you run ls -lh on the /bin and linux-x86_64 directories please.
How did you install elastic? Did you use the RPM installer or another method?
Communications between the elasticsearch JVM and the autodetect process is done via named pipes, these pipes are created in /tmp. In the log there is a message:
Caused by: java.io.FileNotFoundException: /tmp/autodetect_http_response_test_log_17488 (No such file or directory)
/tmp must be readable, if it is mounted with no_exec then autodetect cannot start.
/tmp is the default temporary directory the JVM uses but you can change this with the JVM option -Djava.io.tmpdir=/var/data/elasticsearch/tmp
Can you try adding that setting to ES_HOME/config/jvm.options without a suitable choice of tmpdir. This will required a restart of elasticsearch I'm afraid.
There isn't anything special about mount tmp dir. It's mounted with all the default permissions for Centos 7. drwxrwxrwt. 14 root root 4096 Aug 30 13:37 tmp
I noticed from the directory listing that autodetect was installed 10th August. Did ML work for you previously and then stopped or is that you hadn't tried ML before?
Machine Learning starts a program called controller when elasticsearch starts up this manages the actual processes that do the analytics. Can you check if controller is running please with ps -ef | grep controller
If it is please run ls -l /proc/<controller-process-pid>/ where controller-process-pid is the PID of the process you found with the ps command.
From the grep output elastic+ 17563 17488 0 Jul31 ? 00:00:00 /usr/share/elasticsearch/plugins/x-pack/platform/linux-x86_64/bin/controller
The start time is 31st July this is before the timestamp of the files in the linux-x86_64/bin directory
controller's current work directory seems to have been deleted: lrwxrwxrwx 1 elasticsearch elasticsearch 0 Aug 31 08:12 cwd -> /usr/share/elasticsearch/plugins/x-pack/platform/linux-x86_64/bin (deleted)
and the same for the image lrwxrwxrwx 1 elasticsearch elasticsearch 0 Aug 31 08:12 exe -> /usr/share/elasticsearch/plugins/x-pack/platform/linux-x86_64/bin/controller (deleted)
It looks like the upgrade process did not go cleanly and the old controller did not shut down properly. Can you try shutting down elasticsearch, killing controller kill 17563 and restarting elasticsearch please.
How do you perform the upgrade? That information would help me in tracking down the cause of this issue.
That was it. I restarted all the nodes and it seemed to clear up. The install is managed through puppet, that was the only thing that didn't work right.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.