OS upgrade removed my ES Install

Elastic Stack Elasticsearch
1

Good Day,

We currently run ES 5.x ( we are in the process of upgrading ) but we were running our install on an Ubuntu 16.04 server. So the team went and upgraded the OS of the server to 18.04 and when it was said and done, our ES install was gone??? Why would this happen? and whats the best path forward.

2

I'm thinking it should not have happened, unless instead of an upgrade somebody did a reinstallation.

Check to see if your elasticsearch.yml file (probably in /etc/elasticsearch) and/or your data directory (check elasticsearch.yml for path.data) are still available. If they're still there you might be able to reinstall Elasticsearch, and with any luck it will use the existing elasticsearch.yml and jvm.options files.

Your problem brings up a question: Are you running your Elasticsearch on a single server? If yes, you might want to give serious consideration to implementing an Elasticsearch cluster of at least three servers. If you have a cluster of three or more and something catastrophic happens to one of the nodes (for example, somebody blows it up with a reinstallation of the OS vs an update), the cluster will remain functional and allow a less stressful rebuild of the dead server.

Bob

3

This is a clustered environment. We were able to roll back to a backup, and got everything going again, but it was weird that it happened. So what files should I back up if this happen again when we try to upgrade the Ubuntu version?

Thanks,

4

I'd say the highest priority files to back up prior to a server OS upgrade are the Elasticsearch, and if you're using them Kibana and Logstash, config files (the /etc/{Stack component}/{component}.yml files).

As far as backing up the index/data files goes, I'd suggest making snapshots in Elasticsearch before the server OS upgrades start. You might be able to use an external (to Elasticsearch) backup, but only if you shut down the Elasticsearch cluster first. Using Elasticsearch's snapshots will guarantee consistency between the index files, whereas running an external backup on a running cluster will be almost guaranteed to capture indices in an inconsistent state. As Elastic states in https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html: "The only reliable and supported way to back up a cluster is by taking a snapshot. You cannot back up an Elasticsearch cluster by making copies of the data directories of its nodes."

1 Like
5

Thank You for this information. currently, this was very helpful. Ill read this and see how to move forward.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.