Unassigned shards after upgrade to 8.2

Ian_Simpson · May 13, 2022, 11:48pm

I am managing an Elasticsearch cluster deployed to our Azure environment. I am trying to upgrade our test single-node cluster from 7.x to 8.2. I have done the following:

Upgraded to 7.17.3 and confirmed all shards were in working order
Installed Kibana and walked through the upgrade assistant and modified my Elasticsearch.yml as recommended (used the node.roles config item instead of the deprecated node.xxx config items)
Stopped ES using service elasticsearch stop. I did not perform a flush or anything.
Manually installed the Debian 8.2 package
Restarted ES.

I also tried getting Kibana to upgrade, but it won't start, so I'm leaving it be for now. The upgraded ES will start, but 5 of the 62 shards are unassigned. Looking in the ES log I see this error multiple times:

Caused by: java.lang.IllegalArgumentException: cannot change field "attachment.content" from storeTermVector=true to inconsistent storeTermVector=false

I have not tried reindexing records yet, as I would like to avoid reindexing on the production cluster during upgrade, because re-indexing takes weeks. Is there any way to fix this?

warkolm · May 16, 2022, 12:22am

Welcome to our community!

It would be useful if you could please post the entire Elasticsearch log, from startup.

Ian_Simpson · May 16, 2022, 4:07pm

A clean startup log can be found here: https://1drv.ms/u/s!AkKqV4fHPyyeg7dlVkgh5ZyqcRT3qQ?e=6joLFI. Below is the log up to the first error:

[2022-05-16T15:55:01,638][INFO ][o.e.n.Node               ] [tstE-data-0] version[8.2.0], pid[4574], build[default/deb/b174af62e8dd9f4ac4d25875e9381ffe2b9282c5/2022-04-20T10:35:10.180408517Z], OS[Linux/5.4.0-1078-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/18/18+36]
[2022-05-16T15:55:01,642][INFO ][o.e.n.Node               ] [tstE-data-0] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-05-16T15:55:01,643][INFO ][o.e.n.Node               ] [tstE-data-0] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-3623577881211805543, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms12g, -Xmx12g, -XX:MaxDirectMemorySize=6442450944, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2022-05-16T15:55:04,168][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [aggs-matrix-stats]
[2022-05-16T15:55:04,169][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [analysis-common]
[2022-05-16T15:55:04,169][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [constant-keyword]
[2022-05-16T15:55:04,169][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [data-streams]
[2022-05-16T15:55:04,169][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [frozen-indices]
[2022-05-16T15:55:04,170][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [ingest-common]
[2022-05-16T15:55:04,170][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [ingest-geoip]
[2022-05-16T15:55:04,170][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [ingest-user-agent]
[2022-05-16T15:55:04,170][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [kibana]
[2022-05-16T15:55:04,170][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [lang-expression]
[2022-05-16T15:55:04,171][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [lang-mustache]
[2022-05-16T15:55:04,171][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [lang-painless]
[2022-05-16T15:55:04,172][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [legacy-geo]
[2022-05-16T15:55:04,172][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [mapper-extras]
[2022-05-16T15:55:04,172][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [mapper-version]
[2022-05-16T15:55:04,172][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [parent-join]
[2022-05-16T15:55:04,173][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [percolator]
[2022-05-16T15:55:04,173][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [rank-eval]
[2022-05-16T15:55:04,173][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [reindex]
[2022-05-16T15:55:04,173][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [repositories-metering-api]
[2022-05-16T15:55:04,173][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [repository-azure]
[2022-05-16T15:55:04,173][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [repository-encrypted]
[2022-05-16T15:55:04,174][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [repository-gcs]
[2022-05-16T15:55:04,174][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [repository-s3]
[2022-05-16T15:55:04,175][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [repository-url]
[2022-05-16T15:55:04,175][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [runtime-fields-common]
[2022-05-16T15:55:04,175][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [search-business-rules]
[2022-05-16T15:55:04,175][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [searchable-snapshots]
[2022-05-16T15:55:04,175][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [snapshot-based-recoveries]
[2022-05-16T15:55:04,176][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [snapshot-repo-test-kit]
[2022-05-16T15:55:04,176][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [spatial]
[2022-05-16T15:55:04,176][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [systemd]
[2022-05-16T15:55:04,176][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [transform]
[2022-05-16T15:55:04,177][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [transport-netty4]
[2022-05-16T15:55:04,177][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [unsigned-long]
[2022-05-16T15:55:04,177][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [vector-tile]
[2022-05-16T15:55:04,177][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [vectors]
[2022-05-16T15:55:04,177][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [wildcard]
[2022-05-16T15:55:04,177][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-aggregate-metric]
[2022-05-16T15:55:04,178][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-analytics]
[2022-05-16T15:55:04,179][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-async]
[2022-05-16T15:55:04,179][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-async-search]
[2022-05-16T15:55:04,179][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-autoscaling]
[2022-05-16T15:55:04,179][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-ccr]
[2022-05-16T15:55:04,180][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-core]
[2022-05-16T15:55:04,180][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-deprecation]
[2022-05-16T15:55:04,180][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-enrich]
[2022-05-16T15:55:04,180][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-eql]
[2022-05-16T15:55:04,180][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-fleet]
[2022-05-16T15:55:04,182][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-graph]
[2022-05-16T15:55:04,182][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-identity-provider]
[2022-05-16T15:55:04,182][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-ilm]
[2022-05-16T15:55:04,182][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-logstash]
[2022-05-16T15:55:04,182][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-ml]
[2022-05-16T15:55:04,182][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-monitoring]
[2022-05-16T15:55:04,183][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-ql]
[2022-05-16T15:55:04,183][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-rollup]
[2022-05-16T15:55:04,184][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-security]
[2022-05-16T15:55:04,184][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-shutdown]
[2022-05-16T15:55:04,184][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-sql]
[2022-05-16T15:55:04,185][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-stack]
[2022-05-16T15:55:04,187][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-text-structure]
[2022-05-16T15:55:04,188][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-voting-only-node]
[2022-05-16T15:55:04,188][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded module [x-pack-watcher]
[2022-05-16T15:55:04,189][INFO ][o.e.p.PluginsService     ] [tstE-data-0] loaded plugin [ingest-attachment]
[2022-05-16T15:55:04,227][INFO ][o.e.e.NodeEnvironment    ] [tstE-data-0] using [1] data paths, mounts [[/datadisks/disk1 (/dev/sda1)]], net usable_space [7.4tb], net total_space [7.8tb], types [ext4]
[2022-05-16T15:55:04,227][INFO ][o.e.e.NodeEnvironment    ] [tstE-data-0] heap size [12gb], compressed ordinary object pointers [true]
[2022-05-16T15:55:04,326][INFO ][o.e.n.Node               ] [tstE-data-0] node name [tstE-data-0], node ID [cOmwfTZCRO-aoWC-7J0j7A], cluster name [c2p-es-test], roles [data, master, ingest]
[2022-05-16T15:55:09,181][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [tstE-data-0] [controller/4894] [Main.cc@123] controller (64 bit): Version 8.2.0 (Build a8c0a88ede0ff2) Copyright (c) 2022 Elasticsearch BV
[2022-05-16T15:55:09,331][INFO ][o.e.x.s.Security         ] [tstE-data-0] Security is enabled
[2022-05-16T15:55:09,784][INFO ][o.e.x.s.a.Realms         ] [tstE-data-0] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-05-16T15:55:09,792][INFO ][o.e.x.s.a.s.FileRolesStore] [tstE-data-0] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-05-16T15:55:11,188][INFO ][o.e.t.n.NettyAllocator   ] [tstE-data-0] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=8mb}]
[2022-05-16T15:55:11,216][INFO ][o.e.i.r.RecoverySettings ] [tstE-data-0] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-05-16T15:55:11,248][INFO ][o.e.d.DiscoveryModule    ] [tstE-data-0] using discovery type [multi-node] and seed hosts providers [settings]
[2022-05-16T15:55:12,442][INFO ][o.e.n.Node               ] [tstE-data-0] initialized
[2022-05-16T15:55:12,442][INFO ][o.e.n.Node               ] [tstE-data-0] starting ...
[2022-05-16T15:55:12,496][INFO ][o.e.x.s.c.f.PersistentCache] [tstE-data-0] persistent cache index loaded
[2022-05-16T15:55:12,498][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [tstE-data-0] deprecation component started
[2022-05-16T15:55:12,678][INFO ][o.e.t.TransportService   ] [tstE-data-0] publish_address {10.220.4.4:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {10.220.4.4:9300}
[2022-05-16T15:55:13,142][INFO ][o.e.b.BootstrapChecks    ] [tstE-data-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-05-16T15:55:13,167][INFO ][o.e.c.c.Coordinator      ] [tstE-data-0] cluster UUID [fEwrh_6STZ6tDkv06jEYkA]
[2022-05-16T15:55:13,272][INFO ][o.e.c.s.MasterService    ] [tstE-data-0] elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {tstE-data-0}{cOmwfTZCRO-aoWC-7J0j7A}{gQ6sqsgWSsqQnVoVHuwZdg}{10.220.4.4}{10.220.4.4:9300}{dim} completing election], term: 32, version: 2720, delta: master node changed {previous [], current [{tstE-data-0}{cOmwfTZCRO-aoWC-7J0j7A}{gQ6sqsgWSsqQnVoVHuwZdg}{10.220.4.4}{10.220.4.4:9300}{dim}]}
[2022-05-16T15:55:13,364][INFO ][o.e.c.s.ClusterApplierService] [tstE-data-0] master node changed {previous [], current [{tstE-data-0}{cOmwfTZCRO-aoWC-7J0j7A}{gQ6sqsgWSsqQnVoVHuwZdg}{10.220.4.4}{10.220.4.4:9300}{dim}]}, term: 32, version: 2720, reason: Publication{term=32, version=2720}
[2022-05-16T15:55:13,407][INFO ][o.e.h.AbstractHttpServerTransport] [tstE-data-0] publish_address {10.220.4.4:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {10.220.4.4:9200}
[2022-05-16T15:55:13,407][INFO ][o.e.n.Node               ] [tstE-data-0] started
[2022-05-16T15:55:13,726][INFO ][o.e.l.LicenseService     ] [tstE-data-0] license [f555b711-625a-463d-92e0-bd27d3857ab7] mode [basic] - valid
[2022-05-16T15:55:13,727][INFO ][o.e.x.s.a.Realms         ] [tstE-data-0] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-05-16T15:55:13,731][INFO ][o.e.g.GatewayService     ] [tstE-data-0] recovered [27] indices into cluster_state
[2022-05-16T15:55:14,838][INFO ][o.e.i.g.DatabaseNodeService] [tstE-data-0] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3623577881211805543/geoip-databases/cOmwfTZCRO-aoWC-7J0j7A/GeoLite2-Country.mmdb.tmp.gz]
[2022-05-16T15:55:14,849][INFO ][o.e.i.g.DatabaseNodeService] [tstE-data-0] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3623577881211805543/geoip-databases/cOmwfTZCRO-aoWC-7J0j7A/GeoLite2-City.mmdb.tmp.gz]
[2022-05-16T15:55:14,851][INFO ][o.e.i.g.DatabaseNodeService] [tstE-data-0] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3623577881211805543/geoip-databases/cOmwfTZCRO-aoWC-7J0j7A/GeoLite2-ASN.mmdb.tmp.gz]
[2022-05-16T15:55:15,410][INFO ][o.e.i.g.DatabaseNodeService] [tstE-data-0] successfully loaded geoip database file [GeoLite2-Country.mmdb]
[2022-05-16T15:55:15,519][INFO ][o.e.i.g.DatabaseNodeService] [tstE-data-0] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
[2022-05-16T15:55:16,376][INFO ][o.e.i.g.DatabaseNodeService] [tstE-data-0] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3623577881211805543/geoip-databases/cOmwfTZCRO-aoWC-7J0j7A/GeoLite2-ASN.mmdb.tmp.gz]
[2022-05-16T15:55:16,389][INFO ][o.e.i.g.GeoIpDownloader  ] [tstE-data-0] successfully downloaded geoip database [GeoLite2-ASN.mmdb]
[2022-05-16T15:55:16,781][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [tstE-data-0] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3623577881211805543/geoip-databases/cOmwfTZCRO-aoWC-7J0j7A/GeoLite2-ASN.mmdb]
[2022-05-16T15:55:16,781][INFO ][o.e.i.g.DatabaseNodeService] [tstE-data-0] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
[2022-05-16T15:55:17,383][INFO ][o.e.i.g.DatabaseNodeService] [tstE-data-0] successfully loaded geoip database file [GeoLite2-City.mmdb]
[2022-05-16T15:55:17,982][WARN ][o.e.i.c.IndicesClusterStateService] [tstE-data-0] [rcpnew2][7] marking and sending shard failed due to [failed recovery]
org.elasticsearch.indices.recovery.RecoveryFailedException: [rcpnew2][7]: Recovery failed on {tstE-data-0}{cOmwfTZCRO-aoWC-7J0j7A}{gQ6sqsgWSsqQnVoVHuwZdg}{10.220.4.4}{10.220.4.4:9300}{dim}{xpack.installed=true, fault_domain=0, update_domain=0}
	at org.elasticsearch.index.shard.IndexShard.lambda$executeRecovery$20(IndexShard.java:3098) [elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:170) [elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoveryListener$6(StoreRecovery.java:383) [elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:170) [elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:465) [elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:86) [elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2239) [elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) [elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.2.0.jar:8.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	... 11 more
Caused by: java.lang.IllegalArgumentException: cannot change field "attachment.content" from storeTermVector=true to inconsistent storeTermVector=false
	at org.apache.lucene.index.FieldInfo.verifySameStoreTermVectors(FieldInfo.java:281) ~[lucene-core-9.1.0.jar:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40]
	at org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:432) ~[lucene-core-9.1.0.jar:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40]
	at org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:366) ~[lucene-core-9.1.0.jar:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40]
	at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263) ~[lucene-core-9.1.0.jar:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40]
	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1117) ~[lucene-core-9.1.0.jar:9.1.0 5b522487ba8e0f1002b50a136817ca037aec9686 - jtibs - 2022-03-16 10:32:40]
	at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2384) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.engine.InternalEngine.createWriter(InternalEngine.java:2372) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:228) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:191) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:14) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1930) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1894) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:461) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:88) ~[elasticsearch-8.2.0.jar:8.2.0]
	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:462) ~[elasticsearch-8.2.0.jar:8.2.0]
	... 8 more

Rich_Sutton · May 21, 2022, 3:56pm

I'm experiencing this same issue.

Ian_Simpson · June 2, 2022, 4:06pm

So is the official ES answer is that upgrades don't work all the time? I suppose the work around involves adding new nodes to the cluster running the latest version and reindex the data specifying that only the latest version nodes should hold the newly indexed data; and then retiring the older version nodes once data is reindexed to new nodes?

warkolm · June 2, 2022, 9:51pm

That's, uh, a bit of a stretch.

@DavidTurner do you have ideas here? I can't see the error in GitHub, but it's odd we'd be failing here and it seems related to term vectors? Deleting the index might help, but it's not really a fix.

DavidTurner · June 3, 2022, 8:09am

Sounds like a bug to me. If you can't find anything like it already on GitHub, please report it there.

system · July 1, 2022, 8:10am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.