Hello all,
I have recently run into an issue where our Graylog Server wasn't able to bring up search results and it seemed that our ElasticSearch service crashed.
When it crashed, I noticed that we were running an older version of ElasticSearch and took the opportunity to upgrade the node.
After following the upgrade instructions (https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html), I attempted to start the service, we run this on CentOS 7, and it failed with an error that I didn't understand at the time. With that said, its also possible that I messed up the upgrade somehow and caused the service to not start successfully.
Either way, I blew away the instance and then installed a fresh ElasticSearch node and started everything up fine. When I went to look in my Graylog Server, though, all of our data from the previous 6ish months was missing. Since then, I have found that data in another folder (I didn't realize the DATA_DIR had been changed originally) but am unsure how to merge that old data back in.
Any help is appreciated.
TL;DR - ElasticSearch crashed, attempted to upgrade and failed, fresh installed, old data isn't showing up in search, need to understand how to get it back in the search history.