Can't start elasticsearch with Ubuntu 16.04

Jamie_McLaughlin · August 10, 2017, 8:47am

Hi. What is happening is what is described by the other posters above.

Specifically, on Ububtu (in my case 16), the elasticsearch init script (ie typing 'systemctl start elasticsearch' from a shell) does not start elasticsearch. It returns '[OK]', but elasticsearch does not start. It gives no error message and leaves no logs. It is therefore very difficult to debug, know where to start, or give any useful feedback.

As others have posted, running this command from a shell will successfully launch elasticsearch for me:

sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch.pid --default.config=/etc/elasticsearch/elasticsearch.yml --default.path.home=/usr/share/elasticsearch --default.path.logs=/var/log/elasticsearch --default.path.data=/var/lib/elasticsearch --default.path.work=/tmp/elasticsearch --default.path.conf=/etc/elasticsearch

So it would seem that the init script is broken. Since it gives no output, it is difficult to debug, but I will attempt to do so today.

My suspicion is that it makes wrong assumptions about its environment, since I'm guessing it works on some distros, for example it worked for you.

warkolm · August 13, 2017, 9:49am

I just tried again using https://app.vagrantup.com/v0rtex/boxes/xenial64 and it worked fine, just needed to set heap to suit the VM.

Jamie_McLaughlin · August 14, 2017, 8:23am

How are you starting elasticsearch? Are you using systemctl?

warkolm · August 14, 2017, 8:39am

Yep.

Jamie_McLaughlin · August 14, 2017, 10:14am

Okay, well, as you can see there are plenty of us with this problem, so clearly something is awry. I am still trying to find time to properly debug the script, but I'll post here when I have.

jasontedor · August 14, 2017, 10:14am

We test installing, starting, stopping, upgrading Elasticsearch, and various other packaging-related functionality in our packaging tests on several distributions, including Ubuntu 16.04. The only way that we will be able to make progress on the troubles being reported here is for someone to provide a clear reproduction of the issue. If you have such a reproduction, please post it here, or open an issue at https://github.com/elastic/elasticsearch.

Jamie_McLaughlin · August 14, 2017, 1:20pm

Thanks. The instructions to reproduce are as Sam Ruby originally said:

From a fresh install of Ubuntu 16.04:

sudo apt-get install elasticsearch

From that, the following does not work, but gives no logs or error messages:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sleep 10
curl http://localhost:9200/

Despite that, the following does work:

sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch &
sleep 10
curl http://localhost:9200/

Perhaps we are installing elasticsearch in an unsupported way? warkolm, how did you install when it worked for you? tar.gz or apt-get?

I could try installing it from a tar.gz as advised here: https://www.elastic.co/guide/en/elasticsearch/reference/current/zip-targz.html

However, in this case, there is no support for systemctl that I can see.

Thanks for your help with this. My colleagues are also finding the same thing, so my suspicion is that installing via sudo apt-get install elasticsearch is unsupported?

jasontedor · August 14, 2017, 1:44pm

Can you please clarify one thing. You say:

Are you leaving steps out here, or you are indeed installing not from the Elasticsearch provided repository? We do not support the images provided by Ubuntu, and it does not surprise me that they are broken. Instead, you need to use our repositories:

We support these, and they work with systemd.

Jamie_McLaughlin · August 14, 2017, 2:01pm

Ah ha! I think this is exactly it, and was more than likely Sam Ruby's problem also.

I wasn't leaving steps out - I really was using the Ubuntu package!

Thanks! I'll try the .deb and get back to you.

Thank you again!

jasontedor · August 14, 2017, 2:04pm

You're welcome!

Jamie_McLaughlin · August 14, 2017, 2:50pm

I can confirm this all now works! I similarly had to adjust the starting heap size down from 2G and that might be something to add to your docs, given that some people will be trying out elasticsearch on a test VM with less than 2G. But otherwise this all worked fine and the problem was that the Ubuntu packaged version has a bugged init script. Use the official elasticsearch repositiries instead. As you said, these are the instructions to follow:

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/deb.html

Thanks again

jasontedor · August 14, 2017, 3:11pm

I'm so glad to hear it works.

We lowered the default starting in 6.0.0 for this reason.

kaustubhhiware · August 31, 2017, 3:06am

After a day of unsuccessful attempts, this solved it all. Thanks !

hron84 · October 24, 2017, 3:19pm

It's cool you provide ElasticSearch packages on your own site, however I think it would be great if you can help Ubuntu package maintainer to have a working package in Ubuntu itself too. I ran into this again and I guessed you already contributed with them to work around these issues but in fact, you aren't and this makes me sad. OpenSource developers should work together.

toszter · January 22, 2018, 9:28pm

I just finished an installation on my micro-cluster and faced several similar issues using Elasticsearch v6.1.2, which surprised me because so much of what's mentioned in this thread is supposed to be resolved.

Regardless of that, I documented a step-by-step resolution on Stack Overflow:

I hope this helps.

warkolm · January 31, 2018, 1:13am

I tried this just now on a 16.04 Vagrant box, with java already installed;

root@ubuntu-xenial:~# apt-get install elasticsearch
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  snap-confine
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 28.4 MB of archives.
After this operation, 31.6 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 elasticsearch all 6.1.3 [28.4 MB]
Fetched 28.4 MB in 20s (1,388 kB/s)
Selecting previously unselected package elasticsearch.
(Reading database ... 112672 files and directories currently installed.)
Preparing to unpack .../elasticsearch_6.1.3_all.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (6.1.3) ...
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up elasticsearch (6.1.3) ...
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for ureadahead (0.100.0-19) ...
root@ubuntu-xenial:~# service elasticsearch start
root@ubuntu-xenial:~# ps -ef|grep java
elastic+  2042     1 99 01:09 ?        00:00:05 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
root      2089  1385  0 01:09 pts/0    00:00:00 grep --color=auto java
root@ubuntu-xenial:~#

And the log;

[2018-01-31T01:09:10,347][INFO ][o.e.n.Node               ] [] initializing ...
[2018-01-31T01:09:10,591][INFO ][o.e.e.NodeEnvironment    ] [723DoqN] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [7.5gb], net total_space [9.6gb], types [ext4]
[2018-01-31T01:09:10,592][INFO ][o.e.e.NodeEnvironment    ] [723DoqN] heap size [1007.3mb], compressed ordinary object pointers [true]
[2018-01-31T01:09:10,594][INFO ][o.e.n.Node               ] node name [723DoqN] derived from node ID [723DoqNJQramXRGOU7x2SQ]; set [node.name] to override
[2018-01-31T01:09:10,594][INFO ][o.e.n.Node               ] version[6.1.3], pid[2042], build[af51318/2018-01-26T18:22:55.523Z], OS[Linux/4.4.0-112-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_161/25.161-b12]
[2018-01-31T01:09:10,594][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-01-31T01:09:11,959][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [aggs-matrix-stats]
[2018-01-31T01:09:11,959][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [analysis-common]
[2018-01-31T01:09:11,959][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [ingest-common]
[2018-01-31T01:09:11,959][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [lang-expression]
[2018-01-31T01:09:11,959][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [lang-mustache]
[2018-01-31T01:09:11,960][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [lang-painless]
[2018-01-31T01:09:11,960][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [mapper-extras]
[2018-01-31T01:09:11,960][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [parent-join]
[2018-01-31T01:09:11,960][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [percolator]
[2018-01-31T01:09:11,961][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [reindex]
[2018-01-31T01:09:11,961][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [repository-url]
[2018-01-31T01:09:11,961][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [transport-netty4]
[2018-01-31T01:09:11,961][INFO ][o.e.p.PluginsService     ] [723DoqN] loaded module [tribe]
[2018-01-31T01:09:11,962][INFO ][o.e.p.PluginsService     ] [723DoqN] no plugins loaded
[2018-01-31T01:09:16,037][INFO ][o.e.d.DiscoveryModule    ] [723DoqN] using discovery type [zen]
[2018-01-31T01:09:17,146][INFO ][o.e.n.Node               ] initialized
[2018-01-31T01:09:17,146][INFO ][o.e.n.Node               ] [723DoqN] starting ...
[2018-01-31T01:09:17,474][INFO ][o.e.t.TransportService   ] [723DoqN] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-01-31T01:09:20,597][INFO ][o.e.c.s.MasterService    ] [723DoqN] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {723DoqN}{723DoqNJQramXRGOU7x2SQ}{Q3wjjCJ0Rkmw6oVPqbPcdg}{127.0.0.1}{127.0.0.1:9300}
[2018-01-31T01:09:20,603][INFO ][o.e.c.s.ClusterApplierService] [723DoqN] new_master {723DoqN}{723DoqNJQramXRGOU7x2SQ}{Q3wjjCJ0Rkmw6oVPqbPcdg}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {723DoqN}{723DoqNJQramXRGOU7x2SQ}{Q3wjjCJ0Rkmw6oVPqbPcdg}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-01-31T01:09:20,649][INFO ][o.e.h.n.Netty4HttpServerTransport] [723DoqN] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-01-31T01:09:20,649][INFO ][o.e.n.Node               ] [723DoqN] started
[2018-01-31T01:09:20,649][INFO ][o.e.g.GatewayService     ] [723DoqN] recovered [0] indices into cluster_state

I know that's very much a "works for me" type response

warkolm · January 31, 2018, 1:16am

Following on from the above post.
I think we need to move to separate threads that address each persons issues rather than this single one here, as this is getting a little convoluted and difficult to troubleshoot.

With that in mind I am going to close this topic off. Please feel free to link to this one for further topics, and if you are still having issues then the more info we can get the better able we are to assist and find a resolution