Can't start Elasticsearch 5.5 service

Hi,

I've recently upgraded an Elasticsearch node from 5.4.3 to 5.5 using the windows msi but unfortunately the service will not start. The service manager returns an error stating that the application terminated unexpectedly, and in eventviewer I see the following error from .NET runtime:

Application: elasticsearch.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Elastic.ProcessHosts.Process.StartupException
Stack:
at Elastic.ProcessHosts.Process.ProcessBase.HandleException(System.Exception)
at System.Reactive.Observer1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnError(System.Exception) at System.Reactive.Linq.ObservableImpl.AsObservable1+_[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnError(System.Exception)
at System.Reactive.AutoDetachObserver1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnErrorCore(System.Exception) at Elastic.ProcessHosts.Process.ObservableProcess+<>c__DisplayClass22_0.<CreateProcessExitSubscription>b__0(System.Reactive.EventPattern1<System.Object>)
at System.Reactive.AnonymousSafeObserver`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].OnNext(System.__Canon)
at System.EventHandler.Invoke(System.Object, System.EventArgs)
at System.Diagnostics.Process.RaiseOnExited()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(System.Object, Boolean)

If I attempt to run elasticsearch.exe manually from a command prompt, it returns the error "Unhandled exception: ", and the above log appears in the event viewer.

I also get a report.wer file generated:

Version=1
EventType=CLR20r3
EventTime=131441505567291907
ReportType=2
Consent=1
ReportIdentifier=959b2005-654d-11e7-80d2-00505687b7b7
IntegratorReportIdentifier=959b2004-654d-11e7-80d2-00505687b7b7
NsAppName=elasticsearch.exe
Response.type=4
Sig[0].Name=Problem Signature 01
Sig[0].Value=elasticsearch.exe
Sig[1].Name=Problem Signature 02
Sig[1].Value=5.5.0.0
Sig[2].Name=Problem Signature 03
Sig[2].Value=5956e851
Sig[3].Name=Problem Signature 04
Sig[3].Value=Elastic.ProcessHosts
Sig[4].Name=Problem Signature 05
Sig[4].Value=1.0.0.0
Sig[5].Name=Problem Signature 06
Sig[5].Value=5956e850
Sig[6].Name=Problem Signature 07
Sig[6].Value=57
Sig[7].Name=Problem Signature 08
Sig[7].Value=24
Sig[8].Name=Problem Signature 09
Sig[8].Value=4CY0JEX2JKCFD2IPWRSCWFYWSHBZ0QKL
DynamicSig[1].Name=OS Version
DynamicSig[1].Value=6.3.9600.2.0.0.272.7
DynamicSig[2].Name=Locale ID
DynamicSig[2].Value=2057
DynamicSig[22].Name=Additional Information 1
DynamicSig[22].Value=490f
DynamicSig[23].Name=Additional Information 2
DynamicSig[23].Value=490f32290e491737d5a1b537ac6fa48f
DynamicSig[24].Name=Additional Information 3
DynamicSig[24].Value=f4f2
DynamicSig[25].Name=Additional Information 4
DynamicSig[25].Value=f4f2b51e08961ca8decad8fbcc0511cd
UI[2]=C:\elasticsearch-5.5\bin\elasticsearch.exe
UI[5]=Check online for a solution (recommended)
UI[6]=Check for a solution later (recommended)
UI[7]=Close
UI[8]=Elasticsearch, you know for search! stopped working and was closed
UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
UI[10]=&Close
LoadedModule[0]=C:\elasticsearch-5.5\bin\elasticsearch.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\SYSTEM32\MSCOREE.DLL
LoadedModule[3]=C:\Windows\system32\KERNEL32.dll
LoadedModule[4]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[5]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[6]=C:\Windows\system32\msvcrt.dll
LoadedModule[7]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[8]=C:\Windows\system32\RPCRT4.dll
LoadedModule[9]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscoreei.dll
LoadedModule[10]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[11]=C:\Windows\SYSTEM32\combase.dll
LoadedModule[12]=C:\Windows\system32\USER32.dll
LoadedModule[13]=C:\Windows\system32\GDI32.dll
LoadedModule[14]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
LoadedModule[15]=C:\Windows\SYSTEM32\MSVCR120_CLR0400.dll
LoadedModule[16]=C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\a20cafac04a2e9b3bcb5ec4d674775e5\mscorlib.ni.dll
LoadedModule[17]=C:\Windows\system32\ole32.dll
LoadedModule[18]=C:\Windows\SYSTEM32\kernel.appcore.dll
LoadedModule[19]=C:\Windows\SYSTEM32\CRYPTBASE.dll
LoadedModule[20]=C:\Windows\SYSTEM32\bcryptPrimitives.dll
LoadedModule[21]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clrjit.dll
LoadedModule[22]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[23]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System\6d840e78f9d7168f448d9153f773f5bf\System.ni.dll
LoadedModule[24]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Serv759bfb78#\eec0e0fd7e16b342f2368f29450aa892\System.ServiceProcess.ni.dll
LoadedModule[25]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Core\4ed1137a5fca2b4c293bbc6e37565fa4\System.Core.ni.dll
LoadedModule[26]=C:\Windows\SYSTEM32\CRYPTSP.dll
LoadedModule[27]=C:\Windows\system32\rsaenh.dll
LoadedModule[28]=C:\Windows\SYSTEM32\bcrypt.dll
LoadedModule[29]=C:\Windows\system32\apphelp.dll
LoadedModule[30]=C:\Windows\SYSTEM32\clbcatq.dll
LoadedModule[31]=C:\Windows\SYSTEM32\VERSION.dll
LoadedModule[32]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\diasymreader.dll
FriendlyEventName=Stopped working
ConsentKey=CLR20r3
AppName=Elasticsearch, you know for search!
AppPath=C:\elasticsearch-5.5\bin\elasticsearch.exe
NsPartner=windows
NsGroup=windows8
ApplicationIdentity=2618AD7BF2C2408EB08FF4399DAAE489

Does anyone have any idea how to resolve this?

Thanks

1 Like

Hi ChrisClark,

Many thanks for reporting this issue! While we have automated testing for upgrades between different versions installed by the MSI, the upgrade from a zip based installation has many different initial states to consider.

If you run elasticsearch.exe --debug-env what does it print on the command line?

Other then ES_HOME and ES_CONFIG were any other elasticsearch environment variables set before the installer ran?

Hi Martijn

Many thanks for your incredibly quick response! Here's the result of the command line:

c:\elasticsearch-5.5\bin>elasticsearch.exe --debug-env

Elasticsearch

ES_HOME (in order of precedence)

  • HomeDirectoryProcessVariable = C:\elasticsearch-5.5\
  • HomeDirectoryUserVariable =
  • HomeDirectoryMachineVariable = C:\elasticsearch-5.5\
  • From executable location = c:\elasticsearch-5.5
    ES_CONFIG (in order of precedence)
  • ConfigDirectoryProcessVariable = C:\elasticsearch-5.5\config
  • ConfigDirectoryUserVariable =
  • ConfigDirectoryMachineVariable = C:\elasticsearch-5.5\config
  • Fallback to ES_HOME = C:\elasticsearch-5.5\config

Java

Java paths

  • current = C:\Program Files\Java\jdk1.8.0_60\bin\java.exe
    Java Candidates (in order of precedence)
  • JavaHomeProcessVariable = C:\Program Files\Java\jdk1.8.0_60
  • JavaHomeUserVariable =
  • JavaHomeMachineVariable = C:\Program Files\Java\jdk1.8.0_60
  • JdkRegistry64 =
  • JreRegistry64 =
  • JdkRegistry32 =
  • JreRegistry32 =
    Java checks
  • Using32BitJava = False
  • JAVA_HOME as machine and user variable = False

There was originally the variable that indicated the java heap size - ES_JAVA_OPTS? - but I deleted that prior to starting the installation. However, I did not restart the box and so I think that the variable was still is session, so the first time I attempted to start the service is stopped straight away and logged an error in eventvwr telling me there was a deprecated environment variable. I then restarted the machine and do not see that error now. Perhaps that's caused an issue with the installation?

One other thing to consider is that we've also had version 2.3.4 of Elasticsearch running on this same machine.

Thanks,

Chris

ES_JAVA_OPTS should only prevent the .exe from starting and print a helpful error message, but should have no bearing on the installation.

Do you see any elasticsearch logs being created? If so are you able to share these (microsoft - @ - elastic.co)?

Hi Martijn,

I'm afraid not, the only logs I have from today are from the previous 5.4.3 install.

Is there anywhere else - other than the logging directory I configured during the install - that I might find logs?

Thanks,

Chris

If you run elasticsearch.exe manually from the command line and something unforeseen happens I expect it to print:

An exception occurred while trying to start elasticsearch.

Followed by the exception stack trace.

Can you confirm this is not what happens in your case?

Ah, I tried this earlier and it just said "Unhandled exception: ", but now I get: "Error: Could not create the Java Virtual Machine."

I get nothing in the logs about JVM and the java home path looks correct - any ideas what might cause this?

Thanks,

Chris

Brilliant! atleast the issue bubbles out of the process as expected.

This usually indicates Elasticsearch is configured to take to much memory try lowering the values

-Xmx16296m
-Xms16296m

in

%PROGRAMDATA%\Elastic\Elasticsearch\jvm.options

Would you mind letting me know what they are set to now and how much memory your system has?

Ah! Thanks Martijn, I've got it started now!

Essentially it was my fault. When I went through the installer I forgot to increase the java heap size from the default of 2 up to 3 (this server has 6gb). So I went into the jvm.options file and manually edited it, which I've done before for the 5.4.3 version, but this time I put -Xmx3gb rather than -Xmx3g - now that I've changed it, the service starts up perfectly!

Thanks for your help, apologies for wasting your time!

Thanks,

Chris

1 Like

Not at all! This is potentially another error we could catch and provide a much more helpful error message for.

Glad everything is sorted now :smiley:

2 Likes

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.