# Mistakenly upgraded without reindex or assistant. Now I cannot fix it
**URL:** https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457
**Category:** Elasticsearch
**Created:** [January 9, 2020, 3:26pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457 "2020-01-09T15:26:12Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@brandondash](https://discuss.elastic.co/u/brandondash)
#### Post date: [January 9, 2020, 3:26pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/1 "2020-01-09T15:26:13Z")
We upgraded from 6.4 to 7.4 thinking we didn't have any 5.x indices. It turns out we did. Elasticsearch wouldn't start (by design, documented [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade.html)).
We tried to roll back to 6.4 so we could start elasticsearch to delete the offending indices, but then we ran into this problem:
> org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:66, file:/data/elasticsearch/my\_logs/nodes/0/\_state/node-66.st]
> ....
> Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:36] [node\_meta\_data] unknown field [node\_version], parser not found
That seems to be backed up by a forum post [here](https://discuss.elastic.co/t/unwanted-upgrade-elasticsearch-version/191841/2).
So now I don't know what to do. I can't start elasticsearch in 7.x because of broken indices, and I can't start it in 6.4 because of the error above. Is there any way out of this situation, or did I just lose my entire repo?
That would be... very very bad.
---
### Author:  [@Bernt\_Rostad](https://discuss.elastic.co/u/Bernt_Rostad)
#### Post date: [January 9, 2020, 3:36pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/2 "2020-01-09T15:36:09Z")
> [@brandondash](#):
>
> Is there any way out of this situation, or did I just lose my entire repo?
If you took a snapshot of your indices prior to the upgrade (which is highly recommended before any upgrade) you could simply make a clean install of the 6.4 cluster, start it up and [restore](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-snapshots.html#_restore) the snapshot. When all indices have been restored and the cluster is green, delete the 5.x indices and start the upgrade again.
---
### Author:  [@brandondash](https://discuss.elastic.co/u/brandondash)
#### Post date: [January 9, 2020, 3:38pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/3 "2020-01-09T15:38:17Z")
No snapshot sadly (due to space issues). We did the best we could to mitigate risks by testing in dev and staging environments first, but neither had old indices so we didn't see the problem until it happened.
Any other approaches?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 9, 2020, 3:41pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/4 "2020-01-09T15:41:49Z")
Do you have a copy of the exception and stack trace that the 7.x nodes emitted when they refused to start up? If they noticed the problem before they got too far through the upgrade then it might be recoverable.
---
### Author:  [@brandondash](https://discuss.elastic.co/u/brandondash)
#### Post date: [January 9, 2020, 3:47pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/5 "2020-01-09T15:47:02Z")
I have the stacktrace for each stage of my stumbling and bumbling. Which one are you interested in specifically?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 9, 2020, 4:16pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/6 "2020-01-09T16:16:59Z")
Share as much as you can! Use [https://gist.github.com](https://gist.github.com) since it likely won't fit here.
---
### Author:  [@brandondash](https://discuss.elastic.co/u/brandondash)
#### Post date: [January 9, 2020, 6:27pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/7 "2020-01-09T18:27:39Z")
@DavidTurner [here you go](https://gist.github.com/brandondash/db29d508b6bd3c4376d776b5abb2c889)
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 9, 2020, 6:39pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/8 "2020-01-09T18:39:15Z")
Ok, good, this node stopped in `GatewayMetaState.upgradeMetaData` which looks promising. Is there a file matching `/data/elasticsearch/my_logs/nodes/0/_state/manifest-*.st`?
---
### Author:  [@brandondash](https://discuss.elastic.co/u/brandondash)
#### Post date: [January 9, 2020, 6:44pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/9 "2020-01-09T18:44:56Z")
> [@DavidTurner](#):
>
> Is there a file matching `/data/elasticsearch/my_logs/nodes/0/_state/manifest-*.st` ?
No manifest-\*.st Two files present: global-129.st AND node-68.st
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 9, 2020, 6:50pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/10 "2020-01-09T18:50:32Z")
There aren't many circumstances under which it's ok to manipulate the contents of the data path by hand, but you are in luck that you were upgrading from 6.4 to 7.4.2 and the upgrade stopped at the right place. You should be ok to remove `/data/elasticsearch/my_logs/nodes/0/indices/kDe4rckLScex-zgXJNeOyQ/` which will get rid of the problematic `logstash-2018.08.17` index.
---
### Author:  [@brandondash](https://discuss.elastic.co/u/brandondash)
#### Post date: [January 23, 2020, 7:59pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/11 "2020-01-23T19:59:47Z")
Thanks David. We went through our logs using the strategy you listed and got things up and running.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 20, 2020, 7:59pm UTC](https://discuss.elastic.co/t/mistakenly-upgraded-without-reindex-or-assistant-now-i-cannot-fix-it/214457/12 "2020-02-20T19:59:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.