# Elastic won't start after upgrade to 8.19.4 **URL:** \ **Category:** Elasticsearch\ **Created:** [September 30, 2025, 3:32pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328 "2025-09-30T15:32:16Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![artschooldropout](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@artschooldropout](https://discuss.elastic.co/u/artschooldropout)\ **Post date:** [September 30, 2025, 3:32pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/1 "2025-09-30T15:32:16Z")
We recently attempted to upgrade our cluster from 8.17.4 to 8.19.4 using `apt`. After the upgrade, the service won’t start and we see this line in the logs: ```auto java.lang.IllegalStateException: Failed to parse mappings for index [[.kibana-observability-ai-assistant-kb-000001/UgwoSBEkQPqRj-ItbDSGdA]] ``` and: ```auto Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping: [semantic_text] is available on indices created with 8.11 or higher. ``` Are we toast? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [September 30, 2025, 4:06pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/2 "2025-09-30T16:06:40Z")
> [@artschooldropout](#): > > Are we toast? Well, I would hope not. But did you create snapshot before the upgrade? Can you explain a bit more on current state - the cluster size/topology, how many nodes were upgraded before you saw the error, how (precisely) did you upgrade? ---
**Author:** ![artschooldropout](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@artschooldropout](https://discuss.elastic.co/u/artschooldropout)\ **Post date:** [September 30, 2025, 5:05pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/3 "2025-09-30T17:05:47Z")
We have a two-node cluster running on bare metal. So no snapshots. We issued `apt upgrade` on each node. Edit: I should add that we don’t mind partial data loss. I tried to find the index containing the problematic data, but couldn’t locate it anywhere in the cluster. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [September 30, 2025, 9:00pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/4 "2025-09-30T21:00:50Z")
2-nodes is usually not ideal. I know it's no help now, but FYI you can make snapshots from bare metal too, e.g. to a NFS share. You can at least, now, make a full copy of your data directory, so you can try a few different things on a test system to recover. Are all your indices with at least 1 replica? Did the upgrade complete on both nodes, or just one? > [@artschooldropout](#): > > but couldn’t locate it anywhere in the cluster > ... > java.lang.IllegalStateException: Failed to parse mappings for index [[.kibana-observability-ai-assistant-kb-000001/UgwoSBEkQPqRj-ItbDSGdA]] > > ```auto > > ``` given it gives a specific index with specific uuid, UgwoSBEkQPqRj-ItbDSGdA , it is (or was) surely there at some point? Should be a directory of that name under the indices subdirectory of your data directory on one or both of the Elastic nodes. btw, I thought the Kibana AI Assistant stuff needed a non-basic license, so if you have one you can/should open a support case? ---
**Author:** ![artschooldropout](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@artschooldropout](https://discuss.elastic.co/u/artschooldropout)\ **Post date:** [September 30, 2025, 10:40pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/5 "2025-09-30T22:40:01Z")
Thanks for the information. I tried deleting the offending directory on both nodes. One node actually seems to start up now, but the cluster doesn’t form because quorum hasn’t been reached. I have replicas of all indices. The upgrade completed on both nodes. I don’t have the AI Assistant thing - I don’t know why that is seen here. Is there a way to remove the data on the node that’s complaining about the mapping, join it to the cluster ‘empty’ and replicate data? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [October 1, 2025, 8:38am UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/6 "2025-10-01T08:38:23Z")
My understanding is all your data was replicated, and you have one "good" node that starts up, but can't get fully up as it cannot reach quorum (2), as the other node does not start. The good node, I guess, is just looping waiting for the second node to join? Can you share the actual startup logs from both the "good" and "bad" node please, and the elasticsearch.yml file from both nodes. ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [October 1, 2025, 2:08pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/7 "2025-10-01T14:08:50Z")
It looks like maybe the mapping for the `.kibana-observability-ai-assistant-kb-000001` has been updated to have a `semantic_text` field, but since yours was originally created before 8.11 it can’t be upgraded. ~~This sounds like a bug in the upgrade assistant to me.~~ If possible, I’d restore to 8.17.4 from snapshots, then [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) `.kibana-observability-ai-assistant-kb-000001` so that its creation version is 8.17.4. Then upgrade to 8.19.4. Or if you don’t care about that data you might be able to just delete the index. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [October 1, 2025, 3:41pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/8 "2025-10-01T15:41:50Z")
> [@Keith\_Massey](#): > > I’d restore to 8.17.4 from snapshots They have no snapshots, I asked already. And they deleted the directory that corresponds to the index with "bad" mapping manually (from the filesystem) on both nodes. Their issue now is more that they now have a 2-node cluster that won't start, as one node wont come up and the other node can't reach quorum on its own. btw, I presume @artschooldropout has just done "apt upgrade" on both nodes, maybe a bit carelessly but not expecting this issue for sure, so doubtful they looked at the upgrade assistant prior to the upgrade. Aside: I also noticed I also have a .kibana-observability-ai-assistant-kb-000001 and .kibana-observability-ai-assistant-conversations-000001 indices, and I also have no recollection of ever using "Kibana AI assistant". I think it's fair to say there has been some inflation (some might say bloat) in the "for free" indices that ES clusters are getting in more recent releases. ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [October 1, 2025, 3:55pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/9 "2025-10-01T15:55:02Z")
> [@RainTown](#): > > They have no snapshots, I asked already. Ah, I had missed that. This looks like a bug in the upgrade process though so I’m guessing others are going to hit it and find this thread. In that case, I recommend following the steps from my post. ---
**Author:** ![artschooldropout](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@artschooldropout](https://discuss.elastic.co/u/artschooldropout)\ **Post date:** [October 1, 2025, 4:09pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/10 "2025-10-01T16:09:05Z")
Thanks very much @Keith_Massey and @RainTown for your help on this. We elected to scrap the cluster and start from scratch. In the future we will use snapshots, and follow the official upgrade procedure. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [October 1, 2025, 7:28pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/11 "2025-10-01T19:28:46Z")
> [@Keith\_Massey](#): > thank god I see this thread. I also see these two index running 8.16.1 and I do not use this. both of this index has no documents. will my upgrade to 8.19.4 will fail? ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [October 1, 2025, 7:47pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/12 "2025-10-01T19:47:36Z")
We’ve been investigating this more. It appears right now that it is only a problem if you were on 8.10, and you upgrade (directly or indirectly I believe) to 8.18.7, 8.19.4, 9.0.7, or 9.1.4. If you started on 8.10.0 at any point, I would recommend waiting for detailed instructions before upgrading to 8.19.4. If you can’t wait, definitely make sure you have taken snapshots before upgrading. And before upgrading I would either delete that index, or reindex it. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [October 1, 2025, 7:49pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/13 "2025-10-01T19:49:54Z")
my path is 6.x to many 7.x version to 8.1 to 8.5.3 to 8.16.1 and next step is 8.19.4 ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [October 1, 2025, 7:53pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/14 "2025-10-01T19:53:56Z")
You are probably fine then. I would still highly recommend making sure your snapshots are up to date before upgrading. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [October 1, 2025, 7:56pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/15 "2025-10-01T19:56:24Z")
I will try as cluster is too big can’t fit all the snapshots. 😀 ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [October 1, 2025, 9:12pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/16 "2025-10-01T21:12:01Z")
> [@Keith\_Massey](#): > > It appears right now that it is only a problem if you were on 8.10, and you upgrade (directly or indirectly I believe) to 8.18.7, 8.19.4, 9.0.7, or 9.1.4 First of all, thanks for looking into this issue that @artschooldropout found/raised. And maybe I was unfair to say that he/she may have been "careless", if all they did was hit a specific and unknown (at the time) bug. But, I mused above about the growth in "default indices" that get setup by default, even when you don't use the feature it corresponds too. Consider it feedback that this is quite annoying for some (long-time) users of the core product. And, all that said, @artschooldropout - A 2-node cluster is not a great idea, it adds no real redundancy, quorum is 2, therefore its particularly fragile, ... Please use at least 3 nodes (even if 3rd one is voting-only helps!) in next cluster! And, make snapshots!! ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [October 2, 2025, 7:50pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/17 "2025-10-02T19:50:15Z")
This has been fixed in [Defer Semantic Text Failures on Pre-8.11 Indices by Mikep86 · Pull Request #135845 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/135845) . That will be available in patch releases for all affected versions very soon. Thanks for reporting this @artschooldropout . I’m sorry the fix came too late for you. My understanding is that if you get into the situation where your nodes won’t boot because of the bug in this thread, then you can upgrade to the upcoming patch release and your nodes will start. ---
**Author:** ![artschooldropout](https://avatars.discourse-cdn.com/v4/letter/a/8491ac/32.png) [@artschooldropout](https://discuss.elastic.co/u/artschooldropout)\ **Post date:** [October 3, 2025, 1:40pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/18 "2025-10-03T13:40:20Z")
Sometimes you eat the bear, sometimes the bear eats you. I’m just glad my experience helped prevent this from happening to others. Thanks again for everyone’s help. ---
**Author:** ![Mister\_Alladin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mister_alladin/32/99390_2.png) [@Mister\_Alladin](https://discuss.elastic.co/u/Mister_Alladin)\ **Post date:** [October 6, 2025, 8:19am UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/19 "2025-10-06T08:19:29Z")
Is there anything we can do to fix this issue currently? I moved the AI Index to another location he still complains. Seems to be written somewhere in a mapping table? Or is maybe waiting for 8.14.5 helping? ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [October 6, 2025, 1:05pm UTC](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328/20 "2025-10-06T13:05:55Z")
I’m not sure what you mean by “moved the AI index” but you definitely don’t want to be moving around files on the filesystem if that’s what you mean. The best advice is probably to wait for 8.19.5, which ought to be out _very_ soon. If you can’t wait a day or two, or if upgrading cannot be done quickly for some reason, you can reindex the `.kibana-observability-ai-assistant-kb-000001` index. For example: First reindex `.kibana-observability-ai-assistant-kb-000001` into a new index named `.kibana-observability-ai-assistant-kb-000002`: ```auto POST _reindex { "source": { "index": ".kibana-observability-ai-assistant-kb-000001" }, "dest": { "index": ".kibana-observability-ai-assistant-kb-000002" } } ``` Once that succeeds, delete the original index: ```auto DELETE .kibana-observability-ai-assistant-kb-000001 ``` Re-create the original index: ```auto PUT .kibana-observability-ai-assistant-kb-000001 ``` Now re-index back into the original index (this is all because there is no rename in elasticsarch): ```auto POST _reindex { "source": { "index": ".kibana-observability-ai-assistant-kb-000002" }, "dest": { "index": ".kibana-observability-ai-assistant-kb-000001" } } ``` Check that the alias is still intact: ```auto GET .kibana-observability-ai-assistant-kb/ ``` and ```auto GET _cat/aliases/.kibana-observability-ai-assistant-kb?v ``` If anything is wrong, fix the alias: ```auto POST _aliases { "actions": [ { "add": { "index": ".kibana-observability-ai-assistant-kb-000001", "alias": ".kibana-observability-ai-assistant-kb", "is_write_index": true } } ] } ``` And once everything is good, delete the extra copy of the index: ```auto DELETE .kibana-observability-ai-assistant-kb-000002 ``` [Next page](https://discuss.elastic.co/t/elastic-wont-start-after-upgrade-to-8-19-4/382328.md?page=2)