Elastic lib jars disappear and elasticsearch service stops working

Elastic Stack Elasticsearch
1

Hi,

Whenever elasticsearch service dies with the following lines of logs, all the jar files inside lib folder gone missing. After this, the elasticsearch services stop working and not starting.

Why do elasticsearch jars go missing suddenly??

[INFO ][o.e.c.m.MetaDataDeleteIndexService] [EIcXa59] [myindex/1ozT978aRwyiowvNPmpElQ] deleting index
[INFO ][o.e.n.Node ] [EIcXa59] stopping ...
[INFO ][o.e.x.w.WatcherService ] [EIcXa59] stopping watch service, reason [shutdown initiated]
[INFO ][o.e.x.m.p.l.CppLogMessageHandler] [EIcXa59] [controller/53328] [Main.cc@148] Ml controller exiting
[INFO ][o.e.x.m.p.NativeController] [EIcXa59] Native controller process has stopped - no new native processes can be started
[INFO ][o.e.n.Node ] [EIcXa59] stopped
[INFO ][o.e.n.Node ] [EIcXa59] closing ...
[INFO ][o.e.n.Node ] [EIcXa59] closed

2

Which error? Can't see any in the trace?

Why do elasticsearch jars go missing suddenly??

Elasticsearch does not remove files in lib dir. Did you change any settings? What are they?

3

I don't have anything else suspected other than these lines in the log files..
No settings changed. All are default configurations

4

Which error? Can't see any in the trace?

Can you ls the lib dir?

5

We are running elasticsearch x64 service in windows 10.
It is observed from windows event logs that this issue happens randomly during system restarts.

All jars are missing in lib folder and the only empty folder "tools" is present inside lib

6

What do you observe? I can't see anything.

All jars are missing in lib folder and the only empty folder "tools" is present inside lib

Which lib dir is that? What is the full path to this lib dir?

7

I found below lines in windows event viewer during system restart.

Application popup: elasticsearch-service-x64.exe - Application Error : The instruction at 0x72786bc8 referenced memory at 0x00000188. The memory could not be read.

Click on OK to terminate the program

After reboot, elasticsearch lib folder is empty.

8

Is there anything unclear in the questions I'm asking? Do I have to repeat again my questions? Do you mind answering all of them?

9

Hi @dadoonet,

Environment: OS - Windows 10
Elasticsearch - 6.6.2

After system restart, Elasticsearch service stops working [not starting].
And also found that the jar files in lib folder of elasticsearch are missing. No jars present. only empty tools folder is present in elasticsearch directory.
We are not sure why jars went missing after system reboot.

Below lines are from the elasticsearch.log file after system restart initiated:
[INFO ][o.e.c.m.MetaDataDeleteIndexService] [EIcXa59] [myindex/1ozT978aRwyiowvNPmpElQ] deleting index
[INFO ][o.e.n.Node ] [EIcXa59] stopping ...
[INFO ][o.e.x.w.WatcherService ] [EIcXa59] stopping watch service, reason [shutdown initiated]
[INFO ][o.e.x.m.p.l.CppLogMessageHandler] [EIcXa59] [controller/53328] [Main.cc@148] Ml controller exiting
[INFO ][o.e.x.m.p.NativeController] [EIcXa59] Native controller process has stopped - no new native processes can be started
[INFO ][o.e.n.Node ] [EIcXa59] stopped
[INFO ][o.e.n.Node ] [EIcXa59] closing ...
[INFO ][o.e.n.Node ] [EIcXa59] closed

I do not have any other information to share. Still analysing the issue.

10

Could you try with 6.8.1?

Below lines are from the elasticsearch.log file after system restart initiated:

This line and the next ones are expected as you are shutting down the machine, the service has to stop.

[INFO ][o.e.n.Node ] [EIcXa59] stopping ...

This one is not expected:

[INFO ][o.e.c.m.MetaDataDeleteIndexService] [EIcXa59] [myindex/1ozT978aRwyiowvNPmpElQ] deleting index

Someone decided to remove myindex.

We are not sure why jars went missing after system reboot.

I have no idea. May be you have other processes running on your machine like an antivirus and this one found that the lib dir is not healthy to keep and is moving everything to another location?

But please can you answer this simple question:

What is the full path of the lib dir you are speaking about?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.