Elasticsearch-1.0.0.Beta1-SNAPSHOT problem upgrading lucene 4.4 to lucene 4.5

Elastic Stack Elasticsearch
1

I've been developing against elasticsearch-1.0.0.Beta1-SNAPSHOT. Today I
pulled in recent changes, including the transition to lucence 4.5.
Unfortunately, I seem to have an index that is not forward compatible with
that.

I'm running into the following error

15:44:17.672 [elasticsearch[localnode][generic][T#1]] WARN indices.cluster

  • [localnode] [posts][0] failed to start shard
    org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
    [posts][0] failed recovery
    at
    org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:227)
    ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na]
    at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    ~[na:1.7.0_25]
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    ~[na:1.7.0_25]
    at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
    Caused by: org.elasticsearch.index.engine.EngineCreationFailureException:
    [posts][0] failed to open reader on writer
    at
    org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:290)
    ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na]
    at
    org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:648)
    ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na]
    at
    org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:200)
    ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na]
    at
    org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:174)
    ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na]
    ... 3 common frames omitted
    Caused by: java.io.EOFException: read past EOF:
    NIOFSIndexInput(_0_Disk_0.dvdm in
    path="/Users/jilles/git/localstream/backend/data/localstream-embedded/nodes/0/indices/posts/0/index/_0.cfs"
    slice=140300:140352)
    at
    org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:265)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:51)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at org.apache.lucene.store.DataInput.readVInt(DataInput.java:108)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:218)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer.readNumericEntry(Lucene45DocValuesProducer.java:185)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer.readFields(Lucene45DocValuesProducer.java:120)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.codecs.lucene45.Lucene45DocValuesProducer.(Lucene45DocValuesProducer.java:85)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.codecs.diskdv.DiskDocValuesProducer.(DiskDocValuesProducer.java:31)
    ~[lucene-codecs-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:06:34]
    at
    org.apache.lucene.codecs.diskdv.DiskDocValuesFormat.fieldsProducer(DiskDocValuesFormat.java:56)
    ~[lucene-codecs-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:06:34]
    at
    org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.(PerFieldDocValuesFormat.java:215)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:300)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:142)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at org.apache.lucene.index.SegmentReader.(SegmentReader.java:56)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.index.ReadersAndLiveDocs.getReadOnlyClone(ReadersAndLiveDocs.java:218)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:100)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:379)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:111)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
    ~[lucene-core-4.5.0.jar:4.5.0 1527178 - jpountz - 2013-09-28 14:05:42]
    at
    org.elasticsearch.index.engine.robin.RobinEngine.buildSearchManager(RobinEngine.java:1396)
    ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na]
    at
    org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:280)
    ~[elasticsearch-1.0.0.Beta1-SNAPSHOT.jar:na]
    ... 6 common frames omitted

I've narrowed it down to the exact commit that merged lucene 4.5 on elastic
search master everything after that is broken for me. The commit before
that works fine with a backup index that I have and the commit after that
it just prints stack traces. So 6b02611 is broken for me and a10c195 is
still fine. The way I've tested this is by using a tgz backup of my index
and trying various combinations of versions.

The same index runs fine with 0.90.4 and then upgrading to 0.90.5 works
fine as well (which includes lucene 4.5). From there upgrading to the 1.0
snapshot again breaks my index.

So in summarry in summary, my index seems fine with pre lucene 4.5 master
(a10c195), 0.90.4, and 0.90.5 but is broken with anything after a10c195 on
master.

For now, I'm probably sticking with 0.90.5. I'd very much appreciate any
feedback about what might be wrong on my side or whether I'm running into a
(known?) issue here. I'd be happy to provide more details if needed.

Jilles

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Probably you are too eagerly working way ahead. Just wait for the ES core
team to sort out things on master. Master is unreleased, so you can not
expect it to work always ootb.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

I know the usual disclaimers apply when using master. That being said, I've
been coding against master most of the year without issues and generally
things are actually pretty stable. This specific issue seems weird,
especially since I can narrow it down to one commit and it would be unusual
for things to be in a known broken state for more than a few commits on
master. If something is actually broken, they'd probably want to know.

Jilles

On Tuesday, October 22, 2013 4:40:42 PM UTC+2, Jörg Prante wrote:

Probably you are too eagerly working way ahead. Just wait for the ES core
team to sort out things on master. Master is unreleased, so you can not
expect it to work always ootb.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi Jilles,

On Tue, Oct 22, 2013 at 4:45 PM, Jilles van Gurp jillesvangurp@gmail.comwrote:

I know the usual disclaimers apply when using master. That being said,
I've been coding against master most of the year without issues and
generally things are actually pretty stable. This specific issue seems
weird, especially since I can narrow it down to one commit and it would be
unusual for things to be in a known broken state for more than a few
commits on master. If something is actually broken, they'd probably want to
know.

Indeed master indices don't guarantee backward-compatibility (even though
it is rare that we break it) and the commit that broke your index is not in
Elasticsearch itself but in Lucene: the doc values format that we were
using for _version, called "DiskDocValuesFormat", has been changed in a
non-backward compatible way in Lucene 4.5.0. This was expected since this
format is not officially supported by Lucene.

One thing that the Elasticsearch commit changed however is that _version is
now going to be stored using Lucene45DocValuesFormat, which will be
supported until the last Lucene 5.X release.

The reason why Elasticsearch 0.90.4 and 0.90.5 can still open your index is
that they are based on Lucene 4.4.0. However, these versions are not going
to work fine with your master index since they expect the _version to be
stored in payloads while Elasticsearch master stored the _version in doc
values, so anything relying on versioning may not work.

We will try to announce such backward-compatibility breaks on master in the
future.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Thanks that explains. I mistakenly assumed that 0.90.5 included 4.5
already. Digging through the logs it is clear that is not the case. Will
the upcoming 0.90.6 (with lucene 4.5) be able to handle this without errors?

Jilles

On Wednesday, October 23, 2013 12:11:11 PM UTC+2, Adrien Grand wrote:

Hi Jilles,

On Tue, Oct 22, 2013 at 4:45 PM, Jilles van Gurp <jilles...@gmail.com<javascript:>

wrote:

I know the usual disclaimers apply when using master. That being said,
I've been coding against master most of the year without issues and
generally things are actually pretty stable. This specific issue seems
weird, especially since I can narrow it down to one commit and it would be
unusual for things to be in a known broken state for more than a few
commits on master. If something is actually broken, they'd probably want to
know.

Indeed master indices don't guarantee backward-compatibility (even though
it is rare that we break it) and the commit that broke your index is not in
Elasticsearch itself but in Lucene: the doc values format that we were
using for _version, called "DiskDocValuesFormat", has been changed in a
non-backward compatible way in Lucene 4.5.0. This was expected since this
format is not officially supported by Lucene.

One thing that the Elasticsearch commit changed however is that _version
is now going to be stored using Lucene45DocValuesFormat, which will be
supported until the last Lucene 5.X release.

The reason why Elasticsearch 0.90.4 and 0.90.5 can still open your index
is that they are based on Lucene 4.4.0. However, these versions are not
going to work fine with your master index since they expect the _version to
be stored in payloads while Elasticsearch master stored the _version in doc
values, so anything relying on versioning may not work.

We will try to announce such backward-compatibility breaks on master in
the future.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.