Not able to start 6.3 after upgrade from 6.2

Elastic Stack Elasticsearch
1

We just upgraded from 6.2 to 6.3 and are no longer able to start. The error message states that I cannot start as root, and it seems to be a very common issue looking at posts here and using google.

But I just can't figure out where this is coming from, I've checked everything, and nowhere I seem to be using root anymore

I ensured usr/share/elasticsearch, var/lib/elasticsearch and etc/elasticsearch are not on root, but I keep getting the message that I am using it as root. We decided to uninstall and reinstall from scratch on our development machines but no luck, same error again after vanilla installation. The issue is the same on my production machine which is now completely unavailable.

So since the only responses here are 'do not use root', and as far as i can see I do not use root I have following questions :

What permissions and ownership settings are changed by the installer after upgrading from 6.2 to 6.3?

Are there any other folders used by ES apart from the ones mentioned above? If so can someone share these?

If not, is there a way to roll back?

If it helps, we are running on Ubuntu 16.04, and we had Xpack installed

2

Can you show logs?
How did you install things?
How are you starting Elasticsearch?

3

I installed using the standard apt logic, worked since version 5 so nothing fancy or new here. But this time I wasn't able to get neither Elastic or Kibana running anymore

In normal conditions I have systemd logic taking care of this, but neither one started after reboot.

the logs don't show anything more as below error statement when I try to run it from the commandline

my-sudo-account@my_server:~$ sudo /usr/share/elasticsearch/bin/elasticsearch
[2018-06-15T23:29:30,741][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [sqe_node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.0.jar:6.3.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.0.jar:6.3.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:171) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.0.jar:6.3.0]
... 6 more

By modifying all of the owner / group settings I may have modified more than needed, but bottomline is that neither one of the known locations has root as owner or group, so where is this statement comming from ?

I've been trying numerous combinations of my own account (non root) and / or elasticsearch as a user / group combination so not sure if I did not make things worse that way. Currently these are my used combinations :

etc/default/elasticsearch -> elasticsearch:elasticsearch -> 0660
etc/elasticsearch -> elasticsearch:elasticsearch -> 2750
usr/share/elastisearch -> elasticsearch:elasticsearch -> 0755

note that I tried with various combinations apart from the above ones but all keep telling me I shouldn't use root, on 2 different machines. So obviously I'm missing something, just can't figure out what.

4

How are you starting the process?

5

my-sudo-account@my_server:~$ sudo /usr/share/elasticsearch/bin/elasticsearch

So not as root, but with my user account

6

Using sudo means it will run as root.

7

Ok, but if I don't use sudo it gives me 'command not found' , so if you can not run it as sudo and not without, then how do you run it?

And why was I able to do this for the past few years without a problem?

8

Sound like a path issue more than anything else. Have you tried logging out and back in?

You haven't been able to run Elasticsearch as root for quite some time now, I don't know how your system is configured though.

9

typo indeed, I got no permission errors.

So, I fiddled around with the permissions, got some action then and it turned out also my log files and indexed had changes made to the permissions and/or owner settings after the update. Error messages helped me to fix these one by one and once all these were set it started complaining about the fact I had an old Xpack file. Finally after removing that one it started to work again.

It still does not want to run on boot using my previous settings but ok, I can always reconstruct these ones. At least using the command line ES started working again.

Unfortunately, now Kibana is playing nasty. Same issues, permissions and groups were changed all over the place, modified them all again, got complaints about xPack as for ES, was not able to remove using the normal commands but removing manually seemed to fix this and now I can start it also from the command line again.

But, I still can't open it in the browser now. Kibana starts complaining giving below message :

[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.3.0]: routing [null]] :: {"path":"/.kibana/doc/config%3A6.3.0","query":{} ,"statusCode":503,"response":"{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"No shard available for [get [.kibana][doc][config:6.3.0]: routing [null]]"}],"type":"no_shard_available_a ction_exception","reason":"No shard available for [get [.kibana][doc][config:6.3.0]: routing [null]]"},"status":503}"}

Any idea what goes wrong now ?

10

Seems ES is having problems finding some (older?) Xpack logic preventing it from loading at boot. We did use the trial a while ago, were than converted to basic and maybe there is some garbage left now with the new structure ? What are these appenders and where should they be located?

Jun 16 14:57:10 mining-server elasticsearch[6823]: 2018-06-16 14:57:10,014 main ERROR Null object returned for RollingFile in Appenders.
Jun 16 14:57:10 mining-server elasticsearch[6823]: 2018-06-16 14:57:10,014 main ERROR Null object returned for RollingFile in Appenders.
Jun 16 14:57:10 mining-server elasticsearch[6823]: 2018-06-16 14:57:10,015 main ERROR Unable to locate appender "rolling" for logger config "root"
Jun 16 14:57:10 mining-server elasticsearch[6823]: 2018-06-16 14:57:10,015 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
Jun 16 14:57:10 mining-server elasticsearch[6823]: 2018-06-16 14:57:10,016 main ERROR Unable to locate appender "audit_rolling" for logger config "org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail"
Jun 16 14:57:10 mining-server elasticsearch[6823]: 2018-06-16 14:57:10,016 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Jun 16 14:57:10 mining-server elasticsearch[6823]: 2018-06-16 14:57:10,016 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"

Strangely enough it does start fine with the command line, without giving any error message at all

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.