# Elasticsearch-1.0.0.Beta1-SNAPSHOT problem upgrading lucene 4.4 to lucene 4.5 **URL:** \ **Category:** Elasticsearch\ **Created:** [October 22, 2013, 2:29pm UTC](https://discuss.elastic.co/t/elasticsearch-1-0-0-beta1-snapshot-problem-upgrading-lucene-4-4-to-lucene-4-5/14057 "2013-10-22T14:29:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\ **Post date:** [October 22, 2013, 2:29pm UTC](https://discuss.elastic.co/t/elasticsearch-1-0-0-beta1-snapshot-problem-upgrading-lucene-4-4-to-lucene-4-5/14057/1 "2013-10-22T14:29:20Z")
I've been developing against elasticsearch-1.0.0.Beta1-SNAPSHOT. Today I pulled in recent changes, including the transition to lucence 4.5. Unfortunately, I seem to have an index that is not forward compatible with that. I'm running into the following error 15:44:17.672 [elasticsearch[localnode][generic][T#1]] WARN indices.cluster - [localnode] [posts][0] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [posts][0] failed recovery at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:227) ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0\_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0\_25] at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0\_25] Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [posts][0] failed to open reader on writer at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:290) ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na] at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:648) ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na] at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:200) ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na] at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:174) ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na] ... 3 common frames omitted Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(\_0\_Disk\_0.dvdm in path="/Users/jilles/git/localstream/backend/data/localstream-embedded/nodes/0/indices/posts/0/index/\_0.cfs" slice=140300:140352) at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:265) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:51) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.store.DataInput.readVInt(DataInput.java:108) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:218) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer.readNumericEntry(Lucene45DocValuesProducer.java:185) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer.readFields(Lucene45DocValuesProducer.java:120) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer.(Lucene45DocValuesProducer.java:85) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.codecs.diskdv.DiskDocValuesProducer.(DiskDocValuesProducer.java:31) ~[lucene-codecs-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:06:34] at org.apache.lucene.codecs.diskdv.DiskDocValuesFormat.fieldsProducer(DiskDocValuesFormat.java:56) ~[lucene-codecs-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:06:34] at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.(PerFieldDocValuesFormat.java:215) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:300) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:142) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:56) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:100) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:379) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89) ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42] at org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1396) ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na] at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:280) ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na] ... 6 common frames omitted I've narrowed it down to the exact commit that merged lucene 4.5 on elastic search master everything after that is broken for me. The commit before that works fine with a backup index that I have and the commit after that it just prints stack traces. So 6b02611 is broken for me and a10c195 is still fine. The way I've tested this is by using a tgz backup of my index and trying various combinations of versions. The same index runs fine with 0.90.4 and then upgrading to 0.90.5 works fine as well (which includes lucene 4.5). From there upgrading to the 1.0 snapshot again breaks my index. So in summarry in summary, my index seems fine with pre lucene 4.5 master (a10c195), 0.90.4, and 0.90.5 but is broken with anything after a10c195 on master. For now, I'm probably sticking with 0.90.5. I'd very much appreciate any feedback about what might be wrong on my side or whether I'm running into a (known?) issue here. I'd be happy to provide more details if needed. Jilles -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 22, 2013, 2:40pm UTC](https://discuss.elastic.co/t/elasticsearch-1-0-0-beta1-snapshot-problem-upgrading-lucene-4-4-to-lucene-4-5/14057/2 "2013-10-22T14:40:42Z")
Probably you are too eagerly working way ahead. Just wait for the ES core team to sort out things on master. Master is unreleased, so you can not expect it to work always ootb. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\ **Post date:** [October 22, 2013, 2:45pm UTC](https://discuss.elastic.co/t/elasticsearch-1-0-0-beta1-snapshot-problem-upgrading-lucene-4-4-to-lucene-4-5/14057/3 "2013-10-22T14:45:24Z")
I know the usual disclaimers apply when using master. That being said, I've been coding against master most of the year without issues and generally things are actually pretty stable. This specific issue seems weird, especially since I can narrow it down to one commit and it would be unusual for things to be in a known broken state for more than a few commits on master. If something is actually broken, they'd probably want to know. Jilles On Tuesday, October 22, 2013 4:40:42 PM UTC+2, Jörg Prante wrote: > Probably you are too eagerly working way ahead. Just wait for the ES core > team to sort out things on master. Master is unreleased, so you can not > expect it to work always ootb. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [October 23, 2013, 10:11am UTC](https://discuss.elastic.co/t/elasticsearch-1-0-0-beta1-snapshot-problem-upgrading-lucene-4-4-to-lucene-4-5/14057/4 "2013-10-23T10:11:11Z")
Hi Jilles, On Tue, Oct 22, 2013 at 4:45 PM, Jilles van Gurp [jillesvangurp@gmail.com](mailto:jillesvangurp@gmail.com)wrote: > I know the usual disclaimers apply when using master. That being said, > I've been coding against master most of the year without issues and > generally things are actually pretty stable. This specific issue seems > weird, especially since I can narrow it down to one commit and it would be > unusual for things to be in a known broken state for more than a few > commits on master. If something is actually broken, they'd probably want to > know. Indeed master indices don't guarantee backward-compatibility (even though it is rare that we break it) and the commit that broke your index is not in Elasticsearch itself but in Lucene: the doc values format that we were using for \_version, called "DiskDocValuesFormat", has been changed in a non-backward compatible way in Lucene 4.5.0. This was expected since this format is not officially supported by Lucene. One thing that the Elasticsearch commit changed however is that \_version is now going to be stored using Lucene45DocValuesFormat, which will be supported until the last Lucene 5.X release. The reason why Elasticsearch 0.90.4 and 0.90.5 can still open your index is that they are based on Lucene 4.4.0. However, these versions are not going to work fine with your master index since they expect the \_version to be stored in payloads while Elasticsearch master stored the \_version in doc values, so anything relying on versioning may not work. We will try to announce such backward-compatibility breaks on master in the future. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jilles\_van\_Gurp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jilles_van_gurp/32/879_2.png) [@Jilles\_van\_Gurp](https://discuss.elastic.co/u/Jilles_van_Gurp)\ **Post date:** [October 23, 2013, 11:52am UTC](https://discuss.elastic.co/t/elasticsearch-1-0-0-beta1-snapshot-problem-upgrading-lucene-4-4-to-lucene-4-5/14057/5 "2013-10-23T11:52:01Z")
Thanks that explains. I mistakenly assumed that 0.90.5 included 4.5 already. Digging through the logs it is clear that is not the case. Will the upcoming 0.90.6 (with lucene 4.5) be able to handle this without errors? Jilles On Wednesday, October 23, 2013 12:11:11 PM UTC+2, Adrien Grand wrote: > Hi Jilles, > > On Tue, Oct 22, 2013 at 4:45 PM, Jilles van Gurp \<[jilles...@gmail.com](mailto:jilles...@gmail.com)\ > > > wrote: > > > I know the usual disclaimers apply when using master. That being said, > > I've been coding against master most of the year without issues and > > generally things are actually pretty stable. This specific issue seems > > weird, especially since I can narrow it down to one commit and it would be > > unusual for things to be in a known broken state for more than a few > > commits on master. If something is actually broken, they'd probably want to > > know. > > Indeed master indices don't guarantee backward-compatibility (even though > it is rare that we break it) and the commit that broke your index is not in > Elasticsearch itself but in Lucene: the doc values format that we were > using for \_version, called "DiskDocValuesFormat", has been changed in a > non-backward compatible way in Lucene 4.5.0. This was expected since this > format is not officially supported by Lucene. > > One thing that the Elasticsearch commit changed however is that \_version > is now going to be stored using Lucene45DocValuesFormat, which will be > supported until the last Lucene 5.X release. > > The reason why Elasticsearch 0.90.4 and 0.90.5 can still open your index > is that they are based on Lucene 4.4.0. However, these versions are not > going to work fine with your master index since they expect the \_version to > be stored in payloads while Elasticsearch master stored the \_version in doc > values, so anything relying on versioning may not work. > > We will try to announce such backward-compatibility breaks on master in > the future. > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:11am UTC](https://discuss.elastic.co/t/elasticsearch-1-0-0-beta1-snapshot-problem-upgrading-lucene-4-4-to-lucene-4-5/14057/6 "2017-07-06T02:11:02Z")