Elasticsearch service fails to start

[ShilpiRachna]

We are using Elasticsearch version 5.4.1 on Windows 10.

We have an automation that does the following:

1. Installs Java at a known location (JAVA_HOME is always defined in System environment variable)

2. Install Elasticsearch using "elasticsearch-service.bat install"
   Output:
   [Info @12:06:16.709] Installing service : "elasticsearch-service-x64"
   [Info @12:06:16.709] Using JAVA_HOME (64-bit): "C:\Program Files\Java\jre1.8.0_151"
   [Info @12:06:16.709] The service 'elasticsearch-service-x64' has been installed.

3. Start Elasticsearch using "elasticsearch-service.bat start"
   It fails with the following error:
   2017-12-26 12:06:18 Commons Daemon procrun stderr initialized
   The data area passed to a system call is too small.
   Failed to start service

I read that the issue is cause by JAVA_HOME environment variable not being defined as a System variable. I have verified that it is set correctly. In fact, the issue repros even if JAVA_HOME is correctly set even before running the automation and we install JAVA at the specified location.

However, the issue does not repro if Java is already installed.

Also, verified that heap size is set correctly in the JVM.options.

This issue is intermittent. What else should I check / do to fix this issue?

[Martijn_Laarman]

It does sound like JAVA_HOME is not available

Are you able to share the following logs?

logs\elasticsearch-service-x64-stdout.*.log
logs\elasticsearch-service-x64.*.log

Can you share a little bit more what kind of automation is running and how you are verifying JAVA_HOME is set correctly?

[ShilpiRachna]

Hi Martijn,

It is a .Net application that installs Elasticsearch as a pre-requisite. Copying the relevant Code snippet below:

    public void Install(string esHome, IActivityLog logger)
    {
        Action installStatusCallback = () =>
        {
            if (!WindowsServiceManagement.IsServiceInstalled(ComputerInfo.MachineName, ElasticsearchConstants.ElasticsearchServiceName))
            {
                logger.Info($"Waiting for Elasticsearch operation to complete...");
                throw new ElasticsearchOperationTimedoutException($"Elasticsearch operation was timedout.");
            }
        };

        InvokeServiceBatCommand(esHome, logger, "install", installStatusCallback);
    }

    public void Start(string esHome, IActivityLog logger)
    {
        Action startStatusCallback = () =>
        {
            if (!WindowsServiceManagement.IsServiceRunning(ComputerInfo.MachineName, ElasticsearchConstants.ElasticsearchServiceName))
            {
                logger.Info($"Waiting for Elasticsearch operation to complete...");
                throw new ElasticsearchOperationTimedoutException($"Elasticsearch operation was timedout.");
            }
        };
        InvokeServiceBatCommand(esHome, logger, "start", startStatusCallback);
    }

    private void InvokeServiceBatCommand(string esHome, IActivityLog logger, string command, Action statusCallback)
    {
        string commandString = command.ToLowerInvariant();
        Action elasticsearchConfigAction = () =>
        {
            ProcessStartInfo processInfo = new ProcessStartInfo
            {
                FileName = "cmd.exe",
                Arguments = $"/c elasticsearch-service.bat {commandString}",
                WorkingDirectory = Path.Combine(esHome, "bin"),
                UseShellExecute = false
            };

            string javaHome = Environment.GetEnvironmentVariable("JAVA_HOME", EnvironmentVariableTarget.Machine);
            Debug.Assert(!string.IsNullOrEmpty(javaHome));
            logger.Info($"Adding JAVA_HOME to the environment of child process, value: {javaHome}");
            processInfo.Environment["JAVA_HOME"] = javaHome;

            Task<MSVSCfg.ProcessOutput> task = ProcessHandler.RunExeAsync(processInfo, $"elasticsearch-service.bat {commandString}", logger, 
                default(CancellationToken));
            var timeout = TimeSpan.FromMilliseconds(ElasticsearchConstants.EsOperationTimeInMillsecs);

            if (Task.WaitAll(new[] { task }, timeout))
            {
                statusCallback();
            }
            else if (!task.IsCompleted)
            {
                throw new ElasticsearchOperationTimedoutException($"Elasticsearch operation: {command} was timedout.");
            }
        };

        RetryManager retryManager = new RetryManager(ElasticsearchConstants.EsOperationRetryCount, 
            TimeSpan.FromMilliseconds(300000), (ex) => logger.Warning(ex));
        retryManager.Invoke(elasticsearchConfigAction);
    }

In this particular scenario, JAVA_HOME is set even before we start the ES install (we install JAVA at that location at the start of the application). If we retry the setup a second time, it works. So, the JAVA installation (in the previous retry) is also correct.

Error logs:
logs\elasticsearch-service-x64.*.log

logs\elasticsearch-service-x64-stderr.*.log

[pjanzen (Paul Janzen)]

@ShilpiRachna

The data area passed to a system call is too small

I did a round of google and found this thread Failed to start elasticsearch service · Issue #24187 · elastic/elasticsearch · GitHub where they state that the memory setting might be incorrect. Specificly this might be helpfull Failed to start elasticsearch service · Issue #24187 · elastic/elasticsearch · GitHub

I cannot give you anymore info on this as I am not a windows user or admin.