Unable to start elasticsearch 5.4.0 in docker


#1

I am unable to start elasticsearch 5.4.0 in docker. When I start I see the following error:

Let me know what other information I can provide to help troubleshoot this issue.

[2017-05-06T00:50:34,900][ERROR][o.e.b.Bootstrap ] Exception org.elasticsearch.ElasticsearchException: Failed to create native process factories for Machine Learning at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:297) ~[?:?] at org.elasticsearch.xpack.XPackPlugin.createComponents(XPackPlugin.java:288) ~[?:?] at org.elasticsearch.node.Node.lambda$new$7(Node.java:410) ~[elasticsearch-5.4.0.jar:5.4.0] at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_102] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_102] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_102] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_102] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_102] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_102] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_102] at org.elasticsearch.node.Node.<init>(Node.java:412) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0] Caused by: java.io.FileNotFoundException: /var/log/elasticsearch/controller_log_10 (No such file or directory) at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_102] at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_102] at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.8.0_102] at java.io.FileInputStream.<init>(FileInputStream.java:93) ~[?:1.8.0_102] at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:297) ~[?:?] at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:286) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_102] at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:139) ~[?:?] at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:106) ~[?:?] at org.elasticsearch.xpack.ml.job.process.ProcessPipes.connectStreams(ProcessPipes.java:140) ~[?:?] at org.elasticsearch.xpack.ml.job.process.NativeController.<init>(NativeController.java:65) ~[?:?] at org.elasticsearch.xpack.ml.job.process.NativeControllerHolder.getNativeController(NativeControllerHolder.java:50) ~[?:?] at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:287) ~[?:?] ... 21 more

OS: macos Sierra - 10.12.4
docker Version 17.03.1-ce-mac5 (16048)

The docker image I am using has been created by me and runs using busybox versus centos. Everything works fine with versions 1.x, 2.x and upto 5.3.1 of elasticsearch. Not sure what changed with ES 5.4 that it is complaining about the file path. I can confirm that the directory /var/log/elasticsearch has permissions for the user running docker. Elasticsearch process is able to write logs into the same location (thats where I got the error message from)... so not sure why the machine learning process is not starting...

Any pointers/questions to help debug is much appreciated.


(Jason Tedor) #2

Which C library is the base image that you're using base on? If musl or uclibc, that's probably the source of the trouble. You can you try configuring your container so that Elasticsearch starts with xpack.ml.enabled=false and see if it addresses your issue? Alternatively, if not using a glibc based image, try that. I'm afraid that machine learning is not going to start with musl, nor uclibc.


#3

Thank you @jasontedor. I am using busybox 1.24.1 that depends on glibc.

I went ahead and disabled ML in elasticsearch using elasticsearch.yml

xpack.ml.enabled: false

With this change in place, ES 5.4 in docker starts successfully. However, I do see the following error message in my log file.

Thus there are two questions:

  1. Is there a compatibility issue with glibc that is provided inside busybox 1.24.1 and ES 5.4? I am using Java 1.8u102 inside my container.
  2. Why do I see the following error message in my log even though ML is turned OFF? Is this a bug or should the message be logged at WARN/DEBUG level?

[2017-05-06T22:52:58,425][ERROR][o.e.x.m.MachineLearningFeatureSet] Cannot get native code info for Machine Learning
java.io.FileNotFoundException: /var/log/elasticsearch/controller_log_11 (No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_102]
at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_102]
at java.io.FileInputStream.(FileInputStream.java:138) ~[?:1.8.0_102]
at java.io.FileInputStream.(FileInputStream.java:93) ~[?:1.8.0_102]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:297) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:286) ~[x-pack-5.4.0.jar:5.4.0]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_102]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:139) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:106) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.job.process.ProcessPipes.connectStreams(ProcessPipes.java:140) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.job.process.NativeController.(NativeController.java:65) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.job.process.NativeControllerHolder.getNativeController(NativeControllerHolder.java:50) ~[x-pack-5.4.0.jar:5.4.0]
at org.elasticsearch.xpack.ml.MachineLearningFeatureSet.(MachineLearningFeatureSet.java:87) [x-pack-5.4.0.jar:5.4.0]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_102]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_102]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_102]
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:49) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:116) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:825) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:50) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:191) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:183) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:818) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:183) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:173) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:161) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:96) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:43) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.node.Node.(Node.java:490) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.node.Node.(Node.java:242) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:242) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.0.jar:5.4.0]


(Jason Tedor) #4

I am going to move this topic to the X-Pack category and I will ask someone from the ML team to take a look.


(Sophie Chang) #5

Hi

It looks like there may be a missing dependency. Could you please run the following and share the output?

$ $ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/controller --version

$ ldd $ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/controller

Thanks for your feedback on the log message. We do need to log this as an error, and will look to make this more informative.


#6

Here are the outputs

/ $ $ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/controller --version
/opt/elasticsearch-5.4.0/plugins/x-pack/platform/linux-x86_64/bin/controller: error while loading shared libraries: libuuid.so.1: cannot open shared object file: No such file or directory

/ $ ldd $ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/controller
sh: ldd: not found

/ $ uname -a
Linux e32d7546ed98 4.9.13-moby #1 SMP Sat Mar 25 02:48:44 UTC 2017 x86_64 GNU/Linux

It does seem like my busybox install might be missing some libraries. Can you let know what all I need to make this error go away and have ML working again?


(David Roberts) #7

Hi,

Here are the requirements for the C++ part of Machine Learning on Linux:

  • 64 bit Linux on amd64/x86_64 architecture
  • glibc must be the default C library on the system and version 2.9 or above
  • The following system libraries must be available in directories that the OS will automatically search when running an executable:
    • ld-linux-x86-64.so.2
    • libbz2.so.1
    • libc.so.6
    • libcrypt.so.1
    • libdl.so.2
    • libm.so.6
    • libpthread.so.0
    • librt.so.1
    • libuuid.so.1
    • libz.so.1

Clearly from your output you don't have libuuid.so.1. It's possible that there are other missing libraries that are being masked by the first failure. An alternative to ldd is to set the environment variable LD_TRACE_LOADED_OBJECTS=1, so you could try:

LD_TRACE_LOADED_OBJECTS=1 $ES_HOME/plugins/x-pack/platform/linux-x86_64/bin/controller

That might show up more missing libraries.

As for how you get them installed, you'll have to do that research yourself, as Busybox is not one of our supported Linux distributions (see https://www.elastic.co/support/matrix). On a Redhat-based distribution the command would simply be sudo yum install libuuid and on a Debian derivative it would be sudo apt-get install libuuid1.

Regarding the stack trace that gets logged even when Machine Learning is disabled, we'll make a change so that that doesn't happen in a future release. For the time being please just ignore it.


#8

Thank you @droberts195. The following two libraries are missing in my setup

libuuid.so.1 => not found
libbz2.so.1 => not found

I will find a way to get it added and update this thread once I am done to benefit others.

Thank you @jasontedor @sophie_chang @droberts195 for helping me out.


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.