Kibana runs perfectly if started from terminal, refuses connections if started via Runtime.getRuntime().exec()

I'm writing a java application that injects data into Elastic and displays Kibana in an embedded browser window. During initialization, it attempts to start the services for both. The init code for that looks like this:

Logging.logInfo("Starting Elastic service");
try
{
    gElastic = new Elastic();
}
catch (IOException ioEx)
{
    Logging.showError(this, ioEx, "Failed to start Elastic service", "Critical Error");
    exit(EXIT_CRITICAL_FAILURE);
}
while (!gElastic.gReady)
{
    DateTime.wait(DateTime.HALF_SECOND);
}
DateTime.wait(DateTime.ONE_SECOND);
Logging.logInfo("Starting Kibana service");
try
{
    gKibana = new Kibana();
}
catch (IOException ioEx)
{
    Logging.showError(this, ioEx, "Failed to start Kibana service", "Critical Error");
    exit(EXIT_CRITICAL_FAILURE);
}
while (!gKibana.gReady)
{
    DateTime.wait(DateTime.HALF_SECOND);
}
DateTime.wait(DateTime.ONE_SECOND);

The code that handles the services looks like this:

public static final String KIBANA_BASH = String.format("bash %skibana%sbin%skibana", System.getProperty("user.dir"), File.separator, File.separator);

public Process gProc;
public boolean gReady = false;

public Kibana() throws IOException
{
    gProc = Runtime.getRuntime().exec(KIBANA_BASH);
    new Threading(() ->
    {
        try
        {
            FileOutputStream fos = new FileOutputStream(new File(System.getProperty("user.dir") + "kibana.txt"));
            PrintWriter pw = new PrintWriter(fos, true);
            Scanner scan = new Scanner(gProc.getInputStream());
            while (gProc.isAlive() && !gReady)
            {
                if (scan.hasNext())
                {
                    String line = scan.nextLine();
                    pw.println(line);
                    if (line.trim().endsWith("\"http server running\"}"))
                    {
                        gReady = true;
                        Logging.logInfo("Kibana service ready");
                    }
                }
                else
                {
                    DateTime.wait(DateTime.HALF_SECOND);
                }
            }
            scan.close();
            pw.flush();
            pw.close();
            fos.close();
        }
        catch (Exception ex)
        {
            Logging.logError(ex, "");
        }
    });
}

public void close()
{
    if (gProc.isAlive())
    {
        gProc.destroy();
    }
}

The Elastic class is identical to the Kibana one except that it runs /elasticsearch/bin/elasticsearch instead of kibana/bin/kibana.

If I use the terminal to start the two services, they both work exactly as expected. Using the above method to start them results in Elastic working as expected - I'm able to navigate to the page and see the JSON output - but going to localhost:5601 results in failure to connect even using a standard browser.

The PrintWriter that you can see in the code is there to dump the console output into a file, and by performing a diff on the output from the regular console and the java console I know that with the exception of time stamps and process numbers they are exactly identical. This leads me to believe that either something on the java side is blocking connections to Kibana, although I can't imagine why it would allow connections to Elastic in that case, or something about my (default) Kibana config doesn't like being started by java and doesn't show in the logs.

Which version of Elasticsearch/Kibana are you using?

/cc @jbudz @tiagocosta @tylersmalley

@mathwin can you share the stdout of kibana when starting please?

Also can you just try to init with

public static final String KIBANA_BASH = String.format(".%s%skibana%sbin%skibana", System.getProperty("user.dir"), File.separator, File.separator, File.separator);

gProc = Runtime.getRuntime().exec(KIBANA_BASH);

The above code assumes that System.getProperty("user.dir") did not end with a slash!

Version 7.3.0 - downloaded two days ago.

I would post stdout, but this form won't let me post more than 7k characters at a time, which is about 14k characters too short for even one set of stdout. I can post images, but I doubt that a screenshot of a text editor is going to be particularly helpful. This webform also won't just let me upload it as a text file, so you're out of luck.

As far as ensuring that System.getProperty("user.dir") ends with a separator, I'd have to be a complete newbie to not handle that case. Just above the beginning of Elastic execution is the following lines:

// Set paths for some local files
workingDir = workingDir.endsWith(File.separator) ? workingDir : workingDir + File.separator;
System.setProperty("user.dir", workingDir);

@mathwin could you just try to call Runtime.getRuntime().exec with the equivalent for ./bin/kibana ?

Can you just share the last couple of lines of the stdout? Or instead share a link to download the entire stdout in a file.

Cheers

Stdout from a console run:

{"type":"log","@timestamp":"2019-08-13T11:31:56Z","tags":["info","monitoring","kibana-monitoring"],"pid":19007,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2019-08-13T11:31:56Z","tags":["status","plugin:maps@7.3.0","info"],"pid":19007,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-13T11:31:56Z","tags":["reporting","browser-driver","warning"],"pid":19007,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2019-08-13T11:31:56Z","tags":["reporting","warning"],"pid":19007,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2019-08-13T11:31:56Z","tags":["status","plugin:reporting@7.3.0","info"],"pid":19007,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-08-13T11:31:57Z","tags":["warning","task_manager"],"pid":19007,"message":"Task vis_telemetry \"oss_telemetry-vis_telemetry\" failed in attempt to run: [version_conflict_engine_exception] [oss_telemetry-vis_telemetry]: version conflict, required seqNo [158], primary term [41]. current document has seqNo [159] and primary term [41], with { index_uuid=\"QX7FsIh2RFOyotNB87vD6w\" & shard=\"0\" & index=\".kibana_task_manager\" }"}
{"type":"log","@timestamp":"2019-08-13T11:31:57Z","tags":["listening","info"],"pid":19007,"message":"Server running at http://localhost:5601"}
{"type":"log","@timestamp":"2019-08-13T11:31:57Z","tags":["info","http","server","Kibana"],"pid":19007,"message":"http server running"}
{"type":"log","@timestamp":"2019-08-13T11:31:57Z","tags":["status","plugin:spaces@7.3.0","info"],"pid":19007,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

Stdout from a Runtime.getRuntime().exec(KIBANA_BASH) run:

{"type":"log","@timestamp":"2019-08-13T11:52:41Z","tags":["info","monitoring","kibana-monitoring"],"pid":22221,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2019-08-13T11:52:41Z","tags":["status","plugin:maps@7.3.0","info"],"pid":22221,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-13T11:52:41Z","tags":["reporting","browser-driver","warning"],"pid":22221,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2019-08-13T11:52:41Z","tags":["reporting","warning"],"pid":22221,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2019-08-13T11:52:41Z","tags":["status","plugin:reporting@7.3.0","info"],"pid":22221,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-08-13T11:52:42Z","tags":["warning","task_manager"],"pid":22221,"message":"Task vis_telemetry \"oss_telemetry-vis_telemetry\" failed in attempt to run: [version_conflict_engine_exception] [oss_telemetry-vis_telemetry]: version conflict, required seqNo [176], primary term [45]. current document has seqNo [177] and primary term [45], with { index_uuid=\"QX7FsIh2RFOyotNB87vD6w\" & shard=\"0\" & index=\".kibana_task_manager\" }"}
{"type":"log","@timestamp":"2019-08-13T11:52:42Z","tags":["listening","info"],"pid":22221,"message":"Server running at http://localhost:5601"}
{"type":"log","@timestamp":"2019-08-13T11:52:42Z","tags":["info","http","server","Kibana"],"pid":22221,"message":"http server running"}

Full logs here.

@mathwin I will try to analyse the logs to see if I can spot something.

Meanwhile, could you just run a test and try to run Runtime.getRuntime().exec with the equivalent for ./bin/kibana ? you are running the script with bash

If you still have no luck, we might need a small repo with that code where we can debug and reproduce the problem.

Cheers

Modified the bash calls to:

public static final String ELASTIC_START = String.format("%selasticsearch%sbin%selasticsearch", System.getProperty("user.dir"), File.separator, File.separator);
public static final String KIBANA_START = String.format("%skibana%sbin%skibana", System.getProperty("user.dir"), File.separator, File.separator);

Results are the same. Elastic runs correctly but Kibana refuses all connections. Console output for running normally via terminal and running within Java are verbatim.

I wrote the following program as a testbed:

package elastic;

import java.awt.*;
import java.io.File;
import java.util.Scanner;

import javax.swing.*;

/**
 * <h1>Elastic</h1>
 * @since  2019-08-16
 */
public class Elastic extends JFrame
{
    /* #Region Constants */

    private static final long serialVersionUID = 1L;

    /* #Region Globals */

    Process elastic;
    JPanel ePanel;
    JScrollPane eScroll;
    int eGrid = 0;

    Process kibana;
    JPanel kPanel;
    JScrollPane kScroll;
    int kGrid = 0;

    /* #Region Constructors */

    public Elastic()
    {
        final String path = System.getProperty("user.dir").endsWith(File.separator) ? System.getProperty("user.dir") : System.getProperty("user.dir") + File.separator;

        setLayout(new GridBagLayout());

        GridBagConstraints gbc = new GridBagConstraints();

        gbc.fill = GridBagConstraints.BOTH;
        gbc.insets = new Insets(5, 5, 5, 5);
        gbc.weightx = 0.5;
        gbc.weighty = 0.0;

        ePanel = new JPanel();
        ePanel.setLayout(new BoxLayout(ePanel, BoxLayout.Y_AXIS));

        eScroll = new JScrollPane(ePanel, JScrollPane.VERTICAL_SCROLLBAR_AS_NEEDED, JScrollPane.HORIZONTAL_SCROLLBAR_AS_NEEDED);

        JButton bElastic = new JButton("Start Elastic");
        bElastic.addActionListener(event ->
        {
            new Thread(() ->
            {

                if (elastic != null && elastic.isAlive())
                {
                    elastic.destroy();
                    return;
                }

                try
                {
                    elastic = Runtime.getRuntime().exec(String.format("%selasticsearch%sbin%selasticsearch", path, File.separator, File.separator));

                    bElastic.setText("Stop Elastic");

                    Scanner scan = new Scanner(elastic.getInputStream());
                    while (elastic.isAlive())
                    {
                        if (scan.hasNext())
                        {
                            String line = scan.nextLine();
                            System.out.println(line);
                            log(ePanel, eScroll, eGrid++, line);
                        }
                    }
                    scan.close();
                }
                catch (Exception ex)
                {
                    log(ePanel, eScroll, eGrid++, ex.getMessage());
                    for (StackTraceElement ste : ex.getStackTrace())
                    {
                        log(ePanel, eScroll, eGrid++, ste.toString());
                    }
                }
            }).start();
        });

        kPanel = new JPanel();
        kPanel.setLayout(new BoxLayout(kPanel, BoxLayout.Y_AXIS));

        kScroll = new JScrollPane(kPanel, JScrollPane.VERTICAL_SCROLLBAR_AS_NEEDED, JScrollPane.HORIZONTAL_SCROLLBAR_AS_NEEDED);

        JButton bKibana = new JButton("Start Kibana");
        bKibana.addActionListener(event ->
        {
            new Thread(() ->
            {

                if (kibana != null && kibana.isAlive())
                {
                    kibana.destroy();
                    return;
                }

                try
                {
                    kibana = Runtime.getRuntime().exec(String.format("%skibana%sbin%skibana", path, File.separator, File.separator));

                    bKibana.setText("Stop Kibana");

                    Scanner scan = new Scanner(kibana.getInputStream());
                    while (kibana.isAlive())
                    {
                        if (scan.hasNext())
                        {
                            String line = scan.nextLine();
                            System.out.println(line);
                            log(kPanel, kScroll, kGrid++, line);
                        }
                    }
                    scan.close();
                }
                catch (Exception ex)
                {
                    log(kPanel, kScroll, kGrid++, ex.getMessage());
                    for (StackTraceElement ste : ex.getStackTrace())
                    {
                        log(kPanel, kScroll, kGrid++, ste.toString());
                    }
                }
            }).start();
        });

        getContentPane().add(bElastic, gbc);

        gbc.gridy = 1;
        gbc.weighty = 1.0;
        getContentPane().add(eScroll, gbc);

        gbc.gridx = 1;
        getContentPane().add(kScroll, gbc);

        gbc.gridy = 0;
        gbc.weighty = 0.0;
        getContentPane().add(bKibana, gbc);

        setExtendedState(JFrame.MAXIMIZED_BOTH);

        this.addWindowListener(new WindowAdapter()
        {
            @Override
            public void windowClosing(WindowEvent event)
            {
                if (kibana != null)
                {
                    kibana.destroy();
                }
                if (elastic != null)
                {
                    elastic.destroy();
                }
                System.exit(1);
            }
        });

        setVisible(true);
        toFront();
        requestFocus();
    }

    /* #Region Methods */

    private void log(JPanel panel, JScrollPane scroll, int gridy, String text)
    {
        GridBagConstraints gbc = new GridBagConstraints();
        gbc.gridy = gridy;
        panel.add(new JLabel(text), gbc);

        try
        {
            scroll.validate();

            JScrollBar scrollBar = scroll.getVerticalScrollBar();
            if ((double) (scrollBar.getValue() + scrollBar.getHeight()) / (double) scrollBar.getMaximum() > 0.95)
            {
                scrollBar.setValue(scrollBar.getMaximum());
            }
        }
        catch (Exception ex)
        {
        }

        scroll.validate();
        scroll.repaint();
    }

    public static void main(String[] args)
    {
        try
        {
            new Elastic();
        }
        catch (Exception ex)
        {
            ex.printStackTrace();
        }
    }
}

This works exactly as expected, with both Elastic and Kibana accepting connections. I'll spend some more time playing with it in the main application.

I re-wrote the code in the main application to handle running both Elastic and Kibana in the same class and now it works just fine. Didn't make any changes to the code other than placing both in the same file - the bash commands are the same as they were - but for some reason Kibana is happy now. This is probably something that y'all should continue to look into, because there's certain to be something going on under the hood that needs fixing.