Can't start elasticsearch with Ubuntu 16.04

rubys · April 28, 2016, 8:22pm

I'm hoping that somebody can tell me what step I skipped.

From a fresh install of Ubuntu 16.04:

sudo apt-get install elasticsearch

After that, the following should work but doesn't:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sleep 10
curl http://localhost:9200/

Despite that, the following does work:

sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch &
sleep 10
curl http://localhost:9200/

rubys · April 28, 2016, 8:27pm

Additional information:

$ service elasticsearch status
● elasticsearch.service - LSB: Starts elasticsearch
   Loaded: loaded (/etc/init.d/elasticsearch; bad; vendor preset: enabled)
   Active: active (exited) since Thu 2016-04-28 16:03:54 EDT; 22min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1192 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0
    Tasks: 0 (limit: 512)

Apr 28 16:03:54 rubix2 systemd[1]: Starting LSB: Starts elasticsearch...
Apr 28 16:03:54 rubix2 systemd[1]: Started LSB: Starts elasticsearch.

rubys · April 28, 2016, 8:38pm

No logs are produced by the service

root@rubix2:/usr/share/elasticsearch# ls -l logs
lrwxrwxrwx 1 root root 22 Dec 24 00:15 logs -> /var/log/elasticsearch

root@rubix2:/usr/share/elasticsearch# cd /var/log/elasticsearch/
root@rubix2:/var/log/elasticsearch# rm *
root@rubix2:/var/log/elasticsearch# ls
root@rubix2:/var/log/elasticsearch# service elasticsearch start
root@rubix2:/var/log/elasticsearch# ls
root@rubix2:/var/log/elasticsearch# service elasticsearch status
● elasticsearch.service - LSB: Starts elasticsearch
   Loaded: loaded (/etc/init.d/elasticsearch; bad; vendor preset: enabled)
   Active: active (exited) since Thu 2016-04-28 16:03:54 EDT; 32min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1192 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0
    Tasks: 0 (limit: 512)

Apr 28 16:03:54 rubix2 systemd[1]: Starting LSB: Starts elasticsearch...
Apr 28 16:03:54 rubix2 systemd[1]: Started LSB: Starts elasticsearch.
Apr 28 16:36:19 rubix2 systemd[1]: Started LSB: Starts elasticsearch.
root@rubix2:/var/log/elasticsearch# ls
root@rubix2:/var/log/elasticsearch#

nik9000 · April 28, 2016, 9:18pm

Did you setup the repository like in the instructions here?

What version do you have installed?

rubys · April 28, 2016, 9:37pm

I tried with https://launchpad.net/ubuntu/xenial/+source/elasticsearch, so 1.7.3.

I would think that others are likely to try this, so if that version shouldn't be used, perhaps a blog post or some such is in order? I'll try the latest from packages.elastic.co next (probably tomorrow morning).

rubys · April 29, 2016, 1:28pm

Now I followed the instructions, again from a fresh install of Ubuntu 16.04. Still no success:

root@rubix2:~# dpkg -l elasticsearch
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  elasticsearch  2.3.2        all          Elasticsearch is a distributed RE
root@rubix2:~# /bin/systemctl daemon-reload
root@rubix2:~# /bin/systemctl enable elasticsearch
Synchronizing state of elasticsearch.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable elasticsearch
root@rubix2:~# curl http://localhost:9200/
curl: (7) Failed to connect to localhost port 9200: Connection refused
root@rubix2:~# sleep 10
root@rubix2:~# curl http://localhost:9200/
curl: (7) Failed to connect to localhost port 9200: Connection refused

rubys · April 29, 2016, 1:57pm

Information when run directly:

root@rubix2:/usr/share/elasticsearch# ls /etc/elasticsearch/
elasticsearch.yml  logging.yml  scripts
root@rubix2:/usr/share/elasticsearch# ls
bin  lib  modules  NOTICE.txt  plugins  README.textile
root@rubix2:/usr/share/elasticsearch# bin/elasticsearch
Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config];
Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
    at java.nio.file.Files.walkFileTree(Files.java:2662)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:142)
    at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:103)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:243)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.

eff · May 26, 2016, 8:34am

The int script does work, but only after you enable it in systemd (in disabled/bad state it doesn't use the sysv wrapper).

root@xenial-sandbox:~# systemctl enable elasticsearch
elasticsearch.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install enable elasticsearch
insserv: warning: current start runlevel(s) (empty) of script elasticsearch' overrides LSB defaults (2 3 4 5). insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script elasticsearch' overrides LSB defaults (0 1 6).
root@xenial-sandbox:~# systemctl start elasticsearch
root@xenial-sandbox:~# systemctl status elasticsearch
● elasticsearch.service - LSB: Starts elasticsearch
Loaded: loaded (/etc/init.d/elasticsearch; bad; vendor preset: enabled)
Active: active (running) since Thu 2016-05-26 10:26:36 CEST; 5s ago
Docs: man:systemd-sysv-generator(8)
Process: 1528 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
Tasks: 29 (limit: 512)
Memory: 213.0M
CPU: 4.960s
CGroup: /system.slice/elasticsearch.service
└─1560 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMe

Edit: the 1.7.3 init script refuses to work, i copied the one from the 2.3.1 package, which does.
Edit2: just upgraded systemd to 229-4ubuntu6 and now both init scripts are "broken".

fcoelho · June 1, 2016, 2:47am

It seems that to get Elasticsearch to run on 16.04 you have to set START_DAEMON to true on /etc/default/elasticsearch. It comes commented out by default, and uncommenting it makes Elasticsearch start again just fine.

Be sure to use systemctl restart instead of just start because the service is started right after installation, and apparently there's some socket/pidfile/something that systemd keeps that must be released before being able to start the service again --- I didn't bother to track that down, feel free to look for it

Here's a sample Vagrantfile for it:

Vagrant.configure(2) do |config|
  config.vm.box = 'ubuntu/xenial64'

  config.vm.provision 'shell', inline: <<-SHELL
    sudo apt-get update
    sudo apt-get install elasticsearch -y

    sudo sed -i 's/#START_DAEMON/START_DAEMON/' /etc/default/elasticsearch

    sudo systemctl restart elasticsearch
    systemctl status elasticsearch
  SHELL
end

krisleech · August 11, 2016, 4:07pm

Thank-you this works for me.

cheesepaulo · November 5, 2016, 4:38pm

And the Oscar goes to... @fcoelho

Thank you, there are two days I try to make it work.

YEAH! IT WORKS!

phpremedy · December 26, 2016, 1:46pm

@fcoelho Thanks for the solution.It really works!.

bob-zhao-work · December 29, 2016, 6:48am

Thanks for the script. It really works.
But the guide in online doc is still wrong :
https://www.elastic.co/start

Gunter_Reinitzer · January 17, 2017, 7:28pm

This is still not fixed. On a clean install elasticsearch is not able to start.

agenthampton · April 2, 2017, 4:27pm

This was a pain to trouble shoot. Thanks fcoelho. All my other installs and versions were fine was specific to this apt-get package that it just, no logs, no nothing, said was active, etc, so seemed normal.

askrynnikov · April 9, 2017, 8:54am

IT WORKS!

dev_disola · May 12, 2017, 6:19pm

Works for me too! Thanks =D

bDrwx · June 2, 2017, 7:13pm

I am lucky. I was spent 3 hours.

Jamie_McLaughlin · June 13, 2017, 2:06pm

I'd like my hour back. How can this possibly not be fixed yet? Your default install on probably your most popular platform fails with no error message and no logs. It hardly inspires faith in the platform.

warkolm · August 10, 2017, 4:18am

I just tried the documented method of install and things started without a problem - I only needed to adjust the default heap use cause my VM was small.

If you can provide details on what is happening we can work to fix it, simply saying it's broken makes that very difficult.