I am unable to start elasticsearch 5.4.0 in docker. When I start I see the following error:
Let me know what other information I can provide to help troubleshoot this issue.
[2017-05-06T00:50:34,900][ERROR][o.e.b.Bootstrap ] Exception org.elasticsearch.ElasticsearchException: Failed to create native process factories for Machine Learning at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:297) ~[?:?] at org.elasticsearch.xpack.XPackPlugin.createComponents(XPackPlugin.java:288) ~[?:?] at org.elasticsearch.node.Node.lambda$new$7(Node.java:410) ~[elasticsearch-5.4.0.jar:5.4.0] at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_102] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_102] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_102] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_102] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_102] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_102] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_102] at org.elasticsearch.node.Node.<init>(Node.java:412) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0] Caused by: java.io.FileNotFoundException: /var/log/elasticsearch/controller_log_10 (No such file or directory) at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_102] at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_102] at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.8.0_102] at java.io.FileInputStream.<init>(FileInputStream.java:93) ~[?:1.8.0_102] at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:297) ~[?:?] at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:286) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_102] at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:139) ~[?:?] at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:106) ~[?:?] at org.elasticsearch.xpack.ml.job.process.ProcessPipes.connectStreams(ProcessPipes.java:140) ~[?:?] at org.elasticsearch.xpack.ml.job.process.NativeController.<init>(NativeController.java:65) ~[?:?] at org.elasticsearch.xpack.ml.job.process.NativeControllerHolder.getNativeController(NativeControllerHolder.java:50) ~[?:?] at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:287) ~[?:?] ... 21 more
OS: macos Sierra - 10.12.4
docker Version 17.03.1-ce-mac5 (16048)
The docker image I am using has been created by me and runs using busybox versus centos. Everything works fine with versions 1.x, 2.x and upto 5.3.1 of elasticsearch. Not sure what changed with ES 5.4 that it is complaining about the file path. I can confirm that the directory /var/log/elasticsearch has permissions for the user running docker. Elasticsearch process is able to write logs into the same location (thats where I got the error message from)... so not sure why the machine learning process is not starting...
Any pointers/questions to help debug is much appreciated.
Which C library is the base image that you're using base on? If musl or uclibc, that's probably the source of the trouble. You can you try configuring your container so that Elasticsearch starts with xpack.ml.enabled=false and see if it addresses your issue? Alternatively, if not using a glibc based image, try that. I'm afraid that machine learning is not going to start with musl, nor uclibc.
Thank you @jasontedor. I am using busybox 1.24.1 that depends on glibc.
I went ahead and disabled ML in elasticsearch using elasticsearch.yml
xpack.ml.enabled: false
With this change in place, ES 5.4 in docker starts successfully. However, I do see the following error message in my log file.
Thus there are two questions:
Is there a compatibility issue with glibc that is provided inside busybox 1.24.1 and ES 5.4? I am using Java 1.8u102 inside my container.
Why do I see the following error message in my log even though ML is turned OFF? Is this a bug or should the message be logged at WARN/DEBUG level?
[2017-05-06T22:52:58,425][ERROR][o.e.x.m.MachineLearningFeatureSet] Cannot get native code info for Machine Learning
java.io.FileNotFoundException: /var/log/elasticsearch/controller_log_11 (No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_102]
at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_102]
at java.io.FileInputStream.(FileInputStream.java:138) ~[?:1.8.0_102]
at java.io.FileInputStream.(FileInputStream.java:93) ~[?:1.8.0_102]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:297) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:286) ~[x-pack-5.4.0.jar:5.4.0]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_102]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:139) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:106) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.job.process.ProcessPipes.connectStreams(ProcessPipes.java:140) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.job.process.NativeController.(NativeController.java:65) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.job.process.NativeControllerHolder.getNativeController(NativeControllerHolder.java:50) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.(MachineLearningFeatureSet.java:87) [x-pack-5.4.0.jar:5.4.0]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_102]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_102]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_102]
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:173) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:43) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.node.Node.(Node.java:490) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.node.Node.(Node.java:242) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:242) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0]
/ $ $ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/controller --version
/opt/elasticsearch-5.4.0/plugins/x-pack/platform/linux-x86_64/bin/controller: error while loading shared libraries: libuuid.so.1: cannot open shared object file: No such file or directory
/ $ ldd $ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/controller
sh: ldd: not found
/ $ uname -a
Linux e32d7546ed98 4.9.13-moby #1 SMP Sat Mar 25 02:48:44 UTC 2017 x86_64 GNU/Linux
It does seem like my busybox install might be missing some libraries. Can you let know what all I need to make this error go away and have ML working again?
Here are the requirements for the C++ part of Machine Learning on Linux:
64 bit Linux on amd64/x86_64 architecture
glibc must be the default C library on the system and version 2.9 or above
The following system libraries must be available in directories that the OS will automatically search when running an executable:
ld-linux-x86-64.so.2
libbz2.so.1
libc.so.6
libcrypt.so.1
libdl.so.2
libm.so.6
libpthread.so.0
librt.so.1
libuuid.so.1
libz.so.1
Clearly from your output you don't have libuuid.so.1. It's possible that there are other missing libraries that are being masked by the first failure. An alternative to ldd is to set the environment variable LD_TRACE_LOADED_OBJECTS=1, so you could try:
As for how you get them installed, you'll have to do that research yourself, as Busybox is not one of our supported Linux distributions (see https://www.elastic.co/support/matrix). On a Redhat-based distribution the command would simply be sudo yum install libuuid and on a Debian derivative it would be sudo apt-get install libuuid1.
Regarding the stack trace that gets logged even when Machine Learning is disabled, we'll make a change so that that doesn't happen in a future release. For the time being please just ignore it.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.