Can't start Elasticsearch 5.5 service

ChrisClark · July 10, 2017, 9:14am

Hi,

I've recently upgraded an Elasticsearch node from 5.4.3 to 5.5 using the windows msi but unfortunately the service will not start. The service manager returns an error stating that the application terminated unexpectedly, and in eventviewer I see the following error from .NET runtime:

Application: elasticsearch.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Elastic.ProcessHosts.Process.StartupException
Stack:
at Elastic.ProcessHosts.Process.ProcessBase.HandleException(System.Exception)
at System.Reactive.Observer1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnError(System.Exception) at System.Reactive.Linq.ObservableImpl.AsObservable1+_[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnError(System.Exception)
at System.Reactive.AutoDetachObserver1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnErrorCore(System.Exception) at Elastic.ProcessHosts.Process.ObservableProcess+<>c__DisplayClass22_0.<CreateProcessExitSubscription>b__0(System.Reactive.EventPattern1<System.Object>)
at System.Reactive.AnonymousSafeObserver`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnNext(System.__Canon)
at System.EventHandler.Invoke(System.Object, System.EventArgs)
at System.Diagnostics.Process.RaiseOnExited()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(System.Object, Boolean)

If I attempt to run elasticsearch.exe manually from a command prompt, it returns the error "Unhandled exception: ", and the above log appears in the event viewer.

I also get a report.wer file generated:

Version=1
EventType=CLR20r3
EventTime=131441505567291907
ReportType=2
Consent=1
ReportIdentifier=959b2005-654d-11e7-80d2-00505687b7b7
IntegratorReportIdentifier=959b2004-654d-11e7-80d2-00505687b7b7
NsAppName=elasticsearch.exe
Response.type=4
Sig[0].Name=Problem Signature 01
Sig[0].Value=elasticsearch.exe
Sig[1].Name=Problem Signature 02
Sig[1].Value=5.5.0.0
Sig[2].Name=Problem Signature 03
Sig[2].Value=5956e851
Sig[3].Name=Problem Signature 04
Sig[3].Value=Elastic.ProcessHosts
Sig[4].Name=Problem Signature 05
Sig[4].Value=1.0.0.0
Sig[5].Name=Problem Signature 06
Sig[5].Value=5956e850
Sig[6].Name=Problem Signature 07
Sig[6].Value=57
Sig[7].Name=Problem Signature 08
Sig[7].Value=24
Sig[8].Name=Problem Signature 09
Sig[8].Value=4CY0JEX2JKCFD2IPWRSCWFYWSHBZ0QKL
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.3.9600.2.0.0.272.7
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=2057
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=490f
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=490f32290e491737d5a1b537ac6fa48f
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=f4f2
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=f4f2b51e08961ca8decad8fbcc0511cd
UI[2]=C:\elasticsearch-5.5\bin\elasticsearch.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=Elasticsearch, you know for search! stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=C:\elasticsearch-5.5\bin\elasticsearch.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\SYSTEM32\MSCOREE.DLL
LoadedModule[3]=C:\Windows\system32\KERNEL32.dll
LoadedModule[4]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[5]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[6]=C:\Windows\system32\msvcrt.dll
LoadedModule[7]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[8]=C:\Windows\system32\RPCRT4.dll
LoadedModule[9]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscoreei.dll
LoadedModule[10]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[11]=C:\Windows\SYSTEM32\combase.dll
LoadedModule[12]=C:\Windows\system32\USER32.dll
LoadedModule[13]=C:\Windows\system32\GDI32.dll
LoadedModule[14]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
LoadedModule[15]=C:\Windows\SYSTEM32\MSVCR120_CLR0400.dll
LoadedModule[16]=C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\a20cafac04a2e9b3bcb5ec4d674775e5\mscorlib.ni.dll
LoadedModule[17]=C:\Windows\system32\ole32.dll
LoadedModule[18]=C:\Windows\SYSTEM32\kernel.appcore.dll
LoadedModule[19]=C:\Windows\SYSTEM32\CRYPTBASE.dll
LoadedModule[20]=C:\Windows\SYSTEM32\bcryptPrimitives.dll
LoadedModule[21]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clrjit.dll
LoadedModule[22]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[23]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System\6d840e78f9d7168f448d9153f773f5bf\System.ni.dll
LoadedModule[24]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Serv759bfb78#\eec0e0fd7e16b342f2368f29450aa892\System.ServiceProcess.ni.dll
LoadedModule[25]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Core\4ed1137a5fca2b4c293bbc6e37565fa4\System.Core.ni.dll
LoadedModule[26]=C:\Windows\SYSTEM32\CRYPTSP.dll
LoadedModule[27]=C:\Windows\system32\rsaenh.dll
LoadedModule[28]=C:\Windows\SYSTEM32\bcrypt.dll
LoadedModule[29]=C:\Windows\system32\apphelp.dll
LoadedModule[30]=C:\Windows\SYSTEM32\clbcatq.dll
LoadedModule[31]=C:\Windows\SYSTEM32\VERSION.dll
LoadedModule[32]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\diasymreader.dll
FriendlyEventName=Stopped working
ConsentKey=CLR20r3
AppName=Elasticsearch, you know for search!
AppPath=C:\elasticsearch-5.5\bin\elasticsearch.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=2618AD7BF2C2408EB08FF4399DAAE489

Does anyone have any idea how to resolve this?

Thanks

Martijn_Laarman · July 10, 2017, 10:15am

Hi ChrisClark,

Many thanks for reporting this issue! While we have automated testing for upgrades between different versions installed by the MSI, the upgrade from a zip based installation has many different initial states to consider.

If you run elasticsearch.exe --debug-env what does it print on the command line?

Other then ES_HOME and ES_CONFIG were any other elasticsearch environment variables set before the installer ran?

ChrisClark · July 10, 2017, 10:25am

Hi Martijn

Many thanks for your incredibly quick response! Here's the result of the command line:

c:\elasticsearch-5.5\bin>elasticsearch.exe --debug-env

Elasticsearch

ES_HOME (in order of precedence)

HomeDirectoryProcessVariable = C:\elasticsearch-5.5\
HomeDirectoryUserVariable =
HomeDirectoryMachineVariable = C:\elasticsearch-5.5\
From executable location = c:\elasticsearch-5.5
ES_CONFIG (in order of precedence)
ConfigDirectoryProcessVariable = C:\elasticsearch-5.5\config
ConfigDirectoryUserVariable =
ConfigDirectoryMachineVariable = C:\elasticsearch-5.5\config
Fallback to ES_HOME = C:\elasticsearch-5.5\config

Java

Java paths

current = C:\Program Files\Java\jdk1.8.0_60\bin\java.exe
Java Candidates (in order of precedence)
JavaHomeProcessVariable = C:\Program Files\Java\jdk1.8.0_60
JavaHomeUserVariable =
JavaHomeMachineVariable = C:\Program Files\Java\jdk1.8.0_60
JdkRegistry64 =
JreRegistry64 =
JdkRegistry32 =
JreRegistry32 =
Java checks
Using32BitJava = False
JAVA_HOME as machine and user variable = False

There was originally the variable that indicated the java heap size - ES_JAVA_OPTS? - but I deleted that prior to starting the installation. However, I did not restart the box and so I think that the variable was still is session, so the first time I attempted to start the service is stopped straight away and logged an error in eventvwr telling me there was a deprecated environment variable. I then restarted the machine and do not see that error now. Perhaps that's caused an issue with the installation?

One other thing to consider is that we've also had version 2.3.4 of Elasticsearch running on this same machine.

Thanks,

Chris

Martijn_Laarman · July 10, 2017, 11:10am

ES_JAVA_OPTS should only prevent the .exe from starting and print a helpful error message, but should have no bearing on the installation.

Do you see any elasticsearch logs being created? If so are you able to share these (microsoft - @ - elastic.co)?

ChrisClark · July 10, 2017, 11:14am

Hi Martijn,

I'm afraid not, the only logs I have from today are from the previous 5.4.3 install.

Is there anywhere else - other than the logging directory I configured during the install - that I might find logs?

Thanks,

Chris

Martijn_Laarman · July 10, 2017, 11:47am

If you run elasticsearch.exe manually from the command line and something unforeseen happens I expect it to print:

An exception occurred while trying to start elasticsearch.

Followed by the exception stack trace.

github.com

elastic/windows-installers/blob/5.5/src/ProcessHosts/Elastic.ProcessHosts.Elasticsearch/Program.cs#L45


				if (Environment.UserInteractive)
					Console.Title = $"Elasticsearch {AssemblyVersionInformation.AssemblyInformationalVersion}";
				service.Run();
				var exitCode = service.LastExitCode ?? 0;
				return exitCode;
			}
			catch (Exception e)
			{
				var exitCode = service?.LastExitCode ?? 1;
				if (Environment.UserInteractive)
					e.ToConsole("An exception occurred while trying to start elasticsearch.");
				e.ToEventLog("Elasticsearch");
				return exitCode;
			}
			finally
			{
				service?.Dispose();
				Console.ResetColor();
			}
		}

Can you confirm this is not what happens in your case?

ChrisClark · July 10, 2017, 12:01pm

Ah, I tried this earlier and it just said "Unhandled exception: ", but now I get: "Error: Could not create the Java Virtual Machine."

I get nothing in the logs about JVM and the java home path looks correct - any ideas what might cause this?

Thanks,

Chris

Martijn_Laarman · July 10, 2017, 12:07pm

Brilliant! atleast the issue bubbles out of the process as expected.

This usually indicates elasticsearch is configured to take to much memory try lowering the values

-Xmx16296m
-Xms16296m

in

%PROGRAMDATA%\Elastic\Elasticsearch\jvm.options

Would you mind letting me know what they are set to now and how much memory your system has?

ChrisClark · July 10, 2017, 12:16pm

Ah! Thanks Martijn, I've got it started now!

Essentially it was my fault. When I went through the installer I forgot to increase the java heap size from the default of 2 up to 3 (this server has 6gb). So I went into the jvm.options file and manually edited it, which I've done before for the 5.4.3 version, but this time I put -Xmx3gb rather than -Xmx3g - now that I've changed it, the service starts up perfectly!

Thanks for your help, apologies for wasting your time!

Thanks,

Chris

Martijn_Laarman · July 10, 2017, 12:18pm

Not at all! This is potentially another error we could catch and provide a much more helpful error message for.

Glad everything is sorted now

system · August 7, 2017, 12:18pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elastic search Msi on Server 2016 service cant start Elasticsearch	2	1962	November 3, 2017
Can't start Elasticsearch service, Version: 6.3.2.0 Elasticsearch	2	1515	February 23, 2019
Elasticsearch Service 7.4/7.3 wont start Elasticsearch	3	1522	November 14, 2019
Unable to start ElasticSearch 6.3 Elasticsearch	1	1377	July 16, 2018
ES service keeps crashing Elasticsearch	12	3816	October 30, 2020

Can't start Elasticsearch 5.5 service

c:\elasticsearch-5.5\bin>elasticsearch.exe --debug-env

Elasticsearch

Java

Related topics