Hi,
I have recently upgraded my elastic stack from 5.4.1 to 6.2.1, using puppet to manage the upgrade, on a server in two different environments - development and staging.
The development server upgrade went fine, nothing to report, everything worked as expected.
The upgrade on staging however caused a major issue - All of the indexes created prior to the upgrade disappeared:
curl -XGET http://localhost:9200/logstash-2018.02.27
{"error":{"root_cause":[{"type":"index_not_found_exception","reason": "no such index"....
My indexes were named 'logstash-YYYY.mm.dd' and after the upgrade on development I had to reindex these to the new format we use, which is single type based due to the changes in 6.x, so were now named logstash-<service_name>-YYYY.mm.dd. The reindexing worked fine on my development server.
After the upgrade on staging I decided to check the local files on disk (data dir) and the indices directory was full of sub directories, which was possibly the data? ES was up and running without errors at this point. There was nothing in the log about failing to read anything.
My setup also involves filebeat and logstash so when data started streaming into ES on staging it created the new format indexes fine.
After restarting ES on the staging server (hoping it would fix it?!) the indices directory suddenly dropped to just six sub directories which matched the number of indices that ES was now reporting about - i.e the new ones it created from new data.
Any clue as to what may have happened or how I can diagnose what went wrong? I am not seeing any issues similar to this on the support forum and googling hasn't helped either. Presumably my data has now been lost but I cannot afford for this to happen on a production server.
I can provide logs but obviously they are large so any lines in particular you want me to post I can. My logs do not contain any java exceptions by the way.
Thanks