Liblog4cxx.so.10 change to liblog4cxx.so.10.gz after started a time

Elastic Stack Elasticsearch
1

Elasticsearch version

bin/elasticsearch --version
7.0.0

Plugins installed

bin/elasticsearch-plugin list
analysis-ik

JVM version

java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

OS version

uname -a
Linux log-es05.com 2.6.32-642.6.2.el6.x86_64 #1 SMP Wed Oct 26 06:52:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Related config

node.master: false
node.data: true
node.ingest: true
node.ml: false
xpack.ml.enabled: true

issue

my issue seems strange, i created a cluster with es , when the first time I started it after config , everything seems good , no error. but after a day time (more than 24h but no more thann 48h) , i resatart it , i met the error below .
then i download a new es and try to start , it works well , so i try to compare the diff between the old es and the new-download one .
finaly i find some differ in the dictory elasticsearch-7.0.0/modules/x-pack-ml/platform/linux-x86_64/lib/

the obvious differ is the liblog4cxx.so.10 change to liblog4cxx.so.10.gz
i have done nothing after started

if i replace the old ess dir ```elasticsearch-7.0.0/modules/x-pack-ml/platform/linux-x86_64/lib/``` with the new ess ， it seems every thing ok .

what may be the problem?

1. the dictory in the old es

ll elasticsearch-7.0.0/modules/x-pack-ml/platform/linux-x86_64/lib/
total 19448
-rw-r--r-- 1 deploy deploy  207880 Apr  6 06:57 libapr-1.so.0
-rw-r--r-- 1 deploy deploy  150120 Apr  6 06:57 libaprutil-1.so.0
-rw-r--r-- 1 deploy deploy   71320 Apr  6 06:57 libboost_date_time-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  107176 Apr  6 06:57 libboost_filesystem-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy   91840 Apr  6 06:57 libboost_iostreams-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  537400 Apr  6 06:57 libboost_program_options-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  946584 Apr  6 06:57 libboost_regex-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy   15728 Apr  6 06:57 libboost_system-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  152368 Apr  6 06:57 libboost_thread-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  194344 Apr  6 06:57 libexpat.so.0
-rw-r--r-- 1 deploy deploy   90824 Apr  6 06:57 libgcc_s.so.1
-rw-r--r-- 1 deploy deploy  508114 Apr  6 06:57 liblog4cxx.so.10.gz
-rw-r--r-- 1 deploy deploy 1214432 Apr  6 06:57 libMlApi.so
-rw-r--r-- 1 deploy deploy  841792 Apr  6 06:57 libMlConfig.so
-rw-r--r-- 1 deploy deploy 1210360 Apr  6 06:57 libMlCore.so
-rw-r--r-- 1 deploy deploy 5610136 Apr  6 06:57 libMlMaths.so
-rw-r--r-- 1 deploy deploy 4306904 Apr  6 06:57 libMlModel.so
-rw-r--r-- 1 deploy deploy 2162832 Apr  6 06:57 libstdc++.so.6
-rw-r--r-- 1 deploy deploy 1449256 Apr  6 06:57 libxml2.so.2

2. the dictory in the new es

ll elasticsearch-7.0.0/modules/x-pack-ml/platform/linux-x86_64/lib/
total 20804
-rw-r--r-- 1 deploy deploy  207880 Apr  6 06:57 libapr-1.so.0
-rw-r--r-- 1 deploy deploy  150120 Apr  6 06:57 libaprutil-1.so.0
-rw-r--r-- 1 deploy deploy   71320 Apr  6 06:57 libboost_date_time-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  107176 Apr  6 06:57 libboost_filesystem-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy   91840 Apr  6 06:57 libboost_iostreams-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  537400 Apr  6 06:57 libboost_program_options-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  946584 Apr  6 06:57 libboost_regex-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy   15728 Apr  6 06:57 libboost_system-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  152368 Apr  6 06:57 libboost_thread-gcc73-mt-1_65_1.so.1.65.1
-rw-r--r-- 1 deploy deploy  194344 Apr  6 06:57 libexpat.so.0
-rw-r--r-- 1 deploy deploy   90824 Apr  6 06:57 libgcc_s.so.1
-rw-r--r-- 1 deploy deploy 1898608 Apr  6 06:57 liblog4cxx.so.10
-rw-r--r-- 1 deploy deploy 1214432 Apr  6 06:57 libMlApi.so
-rw-r--r-- 1 deploy deploy  841792 Apr  6 06:57 libMlConfig.so
-rw-r--r-- 1 deploy deploy 1210360 Apr  6 06:57 libMlCore.so
-rw-r--r-- 1 deploy deploy 5610136 Apr  6 06:57 libMlMaths.so
-rw-r--r-- 1 deploy deploy 4306904 Apr  6 06:57 libMlModel.so
-rw-r--r-- 1 deploy deploy 2162832 Apr  6 06:57 libstdc++.so.6
-rw-r--r-- 1 deploy deploy 1449256 Apr  6 06:57 libxml2.so.2

error at restart

[2019-05-22T15:48:53,965][INFO ][o.e.x.s.a.s.FileRolesStore] [ES07] parsed [0] roles from file [/home/deploy/search/elasticsearch-7.0.0/config/roles.yml]
[2019-05-22T15:49:04,442][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [ES07] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to create native process factories for Machine Learning]; nested: FileNotFoundException[/tmp/elasticsearch-2526976195210851989/controller_log_21646 (No such file or directory)];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.0.jar:7.0.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.0.jar:7.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.0.jar:7.0.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]
Caused by: org.elasticsearch.ElasticsearchException: Failed to create native process factories for Machine Learning
	at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:427) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$9(Node.java:438) ~[elasticsearch-7.0.0.jar:7.0.0]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_91]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_91]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_91]
	
Caused by: java.io.FileNotFoundException: /tmp/elasticsearch-2526976195210851989/controller_log_21646 (No such file or directory)
	at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_91]

@richcollier

One guess is that some overzealous log compressor on your system has compressed one of our shared libraries, liblog4cxx.so.10 (presumably because it contains log as part of its name).
This file should NOT be compressed to a .gz file.

this is the answer , thank you richcollier :joy:

2

any body have seen this situation?

3

Hi, friend, i am have same error, but i am installing packaged version 7.1.0. (CentOS 7.6 x86_64) from repository. I read here on the forum that this is because the ML plugin is trying to start its own process, you probably saw this topic. I added the "xpack.ml.enabled: false" parameter to the elasticsearch.yaml file and the error was gone. However, the learning machine doesn't work either. And there was an error caused /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller.
Come on, maybe together we will try to figure it out?
My timezone +3, write, me, please, when you know news?

1 Like
4

i did the same as you :joy:
which topic ?can you give me the link ?
is your problem the same with mine ? that mean
Liblog4cxx.so.10 change to liblog4cxx.so.10.gz after started a time

5

Nothing of interesting..

File liblog4cxx.so.10 exists on my system without changes. I think he is not guilty of anything. I think the system is trying to create something in the /tmp directory for the running Learning Machine, but cannot do it, so the process crashes.
The problem is that I don't know if I need a learning machine or not? :crazy_face:
I am trying other temp-directory with owner "elasticsearch", changing version of Java, but don`t understand, why this "very bad" process cannot running. :rage:

I will patiently study the documentation. :innocent:
If I can understand, I will write here.

6

One guess is that some overzealous log compressor on your system has compressed one of our shared libraries, liblog4cxx.so.10 (presumably because it contains log as part of its name).

This file should NOT be compressed to a .gz file.

1 Like
7

thank you so much , with you remind， i finaly find that it is the bug of our SRE work , i will tell them to deal with it , thank you so much !

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.