Elasticsearch 7.2.0 - Unable to run xpack ml libraries

Version: Elasticsearch 7.2.0
System info: CentOS Linux release 7.6.1810 (Core)

On starting elasticsearch, I'm getting this error log -

[2019-07-14T14:00:26,398][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [fermium] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to create native process factories for Machine Learning]; nested: FileNotFoundException[/tmp/elasticsearch/controller_log_28381 (No such file or directory)];

Elasticsearch Startup options -

ES_TMPDIR=/tmp/elasticsearch ./elasticsearch

I read somewhere that it should have read/write/execute permissions to the /tmp/elasticsearch directory, it has those already -

# ls -lhd /tmp/elasticsearch
drwx------ 3 abhinav2 abhinav2 4.0K Jul 14 14:02 /tmp/elasticsearch

If I disable xpack ml, it works fine: That is, if i add xpack.ml.enabled: false to elasticsearch.yml, it works fine.

Here's the elasticsearch full log -

$ ES_TMPDIR=/tmp/elasticsearch ./elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-07-14T14:00:06,005][INFO ][o.e.e.NodeEnvironment    ] [fermium] using [1] data paths, mounts [[/home (/dev/sda3)]], net usable_space [1.6tb], net total_space [1.7tb], types [ext4]
[2019-07-14T14:00:06,011][INFO ][o.e.e.NodeEnvironment    ] [fermium] heap size [990.7mb], compressed ordinary object pointers [true]
[2019-07-14T14:00:06,015][INFO ][o.e.n.Node               ] [fermium] node name [fermium], node ID [c4pG0chyQRaOSYqBo8pUxQ], cluster name [elasticsearch]
[2019-07-14T14:00:06,016][INFO ][o.e.n.Node               ] [fermium] version[7.2.0], pid[28381], build[default/tar/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/4.9.185-xxxx-std-ipv6-64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
[2019-07-14T14:00:06,017][INFO ][o.e.n.Node               ] [fermium] JVM home [/home/abhinav2/elasticsearch-7.2.0/jdk]
[2019-07-14T14:00:06,017][INFO ][o.e.n.Node               ] [fermium] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/home/abhinav2/elasticsearch-7.2.0, -Des.path.conf=/home/noderig/elasticsearch-7.2.0/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
=== removed due to 7000 word limit ===
[2019-07-14T14:00:14,954][INFO ][o.e.x.s.a.s.FileRolesStore] [fermium] parsed [0] roles from file [/home/abhinav2/elasticsearch-7.2.0/config/roles.yml]
[2019-07-14T14:00:26,398][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [fermium] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to create native process factories for Machine Learning]; nested: FileNotFoundException[/tmp/elasticsearch/controller_log_28381 (No such file or directory)];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]
Caused by: org.elasticsearch.ElasticsearchException: Failed to create native process factories for Machine Learning
	at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:455) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$9(Node.java:438) ~[elasticsearch-7.2.0.jar:7.2.0]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
	at org.elasticsearch.node.Node.<init>(Node.java:441) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]
	... 6 more
=== removed due to 7000 word limit ===

Hello - the easiest way to debug this is to find and execute controller from the prompt:

  • first with --version as a command-line argument ( i.e. /path/to/modules/x-pack-ml/platform/linux-x86_64/bin/controller --version )
  • then again without the --version argument

Report what the output is and post back. Thanks!

[abhinav2@fermium ~]:/home/abhinav2
$ /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller --version
si_signo 4, si_code: 2, si_errno: 0, address: 0x7f271f9609a2, library: /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlCore.so, base: 0x7f271f8b5000, normalized address: 0xab9a2
Illegal instruction

[abhinav2@fermium ~]:/home/abhinav2
$ /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
si_signo 4, si_code: 2, si_errno: 0, address: 0x7fceff8799a2, library: /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlCore.so, base: 0x7fceff7ce000, normalized address: 0xab9a2
Illegal instruction

See: Elasticsearch 7.1.1 Failed to create native process factories for Machine Learning

Looks like you are attempting to run on an older machine that does not have support for the SSE4.2 instruction sets.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.