ElasticSearch service will not stay running after rolling upgrade to 6.2.2-1

[msterne (Mark Sterne)]

So, I'd like to start this off by making perfectly clear that I am very inexperienced with Linux (we have CentOS7) and Elasticsearch. I have inherited a complete ElasticSearch, Logstash and Kibana environment and have been asked to upgrade it and start using it for log monitoring.

The servers were 5.6.2-1 and I upgraded them to 5.6.8-1 before doing a rolling upgrade (took me some time figuring out how to do this with Linux) 6.2.2-1. I completed the upgrade using the rolling upgrade instructions on the website.

Now, when I restart the service systemctl start elasticsearch.service, it shows that the service is started when I go to systemctl status elasticsearch. If I wait a minute and type the same command it states the status is "failed". Referring to main ERROR Null object returned RollingFile in Appenders. Then "unable to locate appender "rolling " for logger config "root".

I'm not familiar with how to copy and paste the results of a command.

So there you have it. I've upgraded 2 of my three servers in the Elastic cluster and have Logstash and Kibana running on another server (3 Elasticsearch servers and a Logstash/Kibana server. 4 in total). I have suspended moving forward with upgrades until I get a little more insight into how to get the service running without stopping.

Any help would be greatly appreciated.

[warkolm (Mark Walkom)]

Have a look at /var/log/elasticsearch/elasticsearch.log, you should be able to highlight the text to copy to clipboard and then paste it in here.

[msterne (Mark Sterne)]

Thanks for your reply Mark. I went to the /var/log/elasticsearch/ directory and I do not see a elasticsearch.log file. I only see gc.log files and production+"date".log files. The most recent production+date.log file is from 2-15-2018.

I do recall making a change to the elasticsearch.service file at one point . I was looking through a topic (you actually commented on it) that was a similar issue after upgrading to from 5.5 to 6.0. At any rate, I changed the ExecStart= portion of the elasticsearch.service file and removed the --quite from the end of the line. I have since re-inserted it. That may be why I'm not logging things correctly.

I'm still unable to highlight and select text to a clipboard. I'm running these servers in a Microsoft Hyper-V environment. I get an error when trying to use the clipboard. I'll check with one of my colleagues about getting that working. I did find something that mentioned gpm as a way to use the mouse to select text. So, I'm looking into that as well.

[msterne (Mark Sterne)]

O.K., I connected via SSH using mRemoteMG and was able to copy the results of the elasticsearch service status...here are the results.

`[root@elastic03 ~]# [root@elastic03 ~]# systemctl status elasticsearch
Docs: http://www.elastic.co
Process: 1042 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 1042 (code=exited, status=1/FAILURE)

Feb 27 08:37:59 elastic03 elasticsearch[1042]: 2018-02-27 08:37:59,636 main ERROR Null object returned for RollingFile in Appenders.
Feb 27 08:37:59 elastic03 elasticsearch[1042]: 2018-02-27 08:37:59,636 main ERROR Null object returned for RollingFile in Appenders.
Feb 27 08:37:59 elastic03 elasticsearch[1042]: 2018-02-27 08:37:59,637 main ERROR Null object returned for RollingFile in Appenders.
Feb 27 08:37:59 elastic03 elasticsearch[1042]: 2018-02-27 08:37:59,637 main ERROR Unable to locate appender "rolling" for logger config "root"
Feb 27 08:37:59 elastic03 elasticsearch[1042]: 2018-02-27 08:37:59,637 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.indexing.slowlog.index"
Feb 27 08:37:59 elastic03 elasticsearch[1042]: 2018-02-27 08:37:59,638 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Feb 27 08:37:59 elastic03 elasticsearch[1042]: 2018-02-27 08:37:59,638 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Feb 27 08:38:00 elastic03 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Feb 27 08:38:00 elastic03 systemd[1]: Unit elasticsearch.service entered failed state.
Feb 27 08:38:00 elastic03 systemd[1]: elasticsearch.service failed.`

May this helps.