Luke not recognizing index from ES git master


(Sebastian Gavarini) #1

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.


(Sebastian Gavarini) #2

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at
org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(SegmentInfos.java:
199)
at
org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
134)
at
org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.


(Lukáš Vlček) #3

I remember I was getting similar error while I was using some older ES
version (0.11.0 or something like that). The point at that time was that I
was not able to open Lucene index from gateway in Luke but I was able to
open the same "index" from work directory. Hope I remember correctly :slight_smile:
But sure many things have changed since then. Local gateway was added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are some
breaking changes compared to 0.12.x version (check
https://github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavarini@gmail.com wrote:

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(SegmentInfos.java:
199)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
134)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.


(Sebastian Gavarini) #4

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older ES
version (0.11.0 or something like that). The point at that time was that I
was not able to open Lucene index from gateway in Luke but I was able to
open the same "index" from work directory. Hope I remember correctly :slight_smile:
But sure many things have changed since then. Local gateway was added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are some
breaking changes compared to 0.12.x version (checkhttps://github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com wrote:

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment Infos.java:
199)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
134)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.


(Sebastian Gavarini) #5

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at
org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(SegmentInfos.java:
199)
at
org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
134)
at org.apache.lucene.index.SegmentInfos
$FindSegmentsFile.run(SegmentInfos.java:571)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older ES
version (0.11.0 or something like that). The point at that time was that I
was not able to open Lucene index from gateway in Luke but I was able to
open the same "index" from work directory. Hope I remember correctly :slight_smile:
But sure many things have changed since then. Local gateway was added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are some
breaking changes compared to 0.12.x version (checkhttps://github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com wrote:

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment Infos.java:
199)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
134)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.


(Sebastian Gavarini) #6

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at
org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment Infos.java:
199)
at
org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
134)
at org.apache.lucene.index.SegmentInfos
$FindSegmentsFile.run(SegmentInfos.java:571)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older ES
version (0.11.0 or something like that). The point at that time was that I
was not able to open Lucene index from gateway in Luke but I was able to
open the same "index" from work directory. Hope I remember correctly :slight_smile:
But sure many things have changed since then. Local gateway was added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are some
breaking changes compared to 0.12.x version (checkhttps://github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com wrote:

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment Infos.java:
199)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
134)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.


(Lukáš Vlček) #7

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
http://lucene.apache.org/java/3_0_2/fileformats.html#File
Naming
<http://lucene.apache.org/java/3_0_2/fileformats.html#File Naming>It is
internal ES file (checksum file) and Lucene will not understand it. (Check
AbstractStore class in ES, method writeChecksum). Not sure why Lucene tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavarini@gmail.com wrote:

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at
    

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
    at org.apache.lucene.index.IndexReader.open(IndexReader.java:188)
    

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older
ES

version (0.11.0 or something like that). The point at that time was
that I

was not able to open Lucene index from gateway in Luke but I was able
to

open the same "index" from work directory. Hope I remember correctly
:slight_smile:

But sure many things have changed since then. Local gateway was
added, index

is not compound now and in recent 0.13.0-SNAPSHOT version there are
some

breaking changes compared to 0.12.x version (checkhttps://
github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note

the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com
wrote:

In fact if I try to open the same data path index created with
0.13.0-

SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"

   at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI
ndexShardGateway.java:

  1. at org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the
Lucene

indices to see if my analyzer configuration was doing the right
thing.

If I use ES 0.12.1 and then open the underlying Lucene index with
Luke

1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from
git

yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.


(Lukáš Vlček) #8

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (see
https://github.com/elasticsearch/elasticsearch/issues/closed/#issue/463)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file: http://lucene.apache.org/java/3_0_2/fileformats.html#File
Naming http://lucene.apache.org/java/3_0_2/fileformats.html#File+Naming
http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt is
internal ES file (checksum file) and Lucene will not understand it. (Check
AbstractStore class in ES, method writeChecksum). Not sure why Lucene tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavarini@gmail.com wrote:

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at
    

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at
    

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older
ES

version (0.11.0 or something like that). The point at that time was
that I

was not able to open Lucene index from gateway in Luke but I was
able to

open the same "index" from work directory. Hope I remember correctly
:slight_smile:

But sure many things have changed since then. Local gateway was
added, index

is not compound now and in recent 0.13.0-SNAPSHOT version there are
some

breaking changes compared to 0.12.x version (checkhttps://
github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note

the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com
wrote:

In fact if I try to open the same data path index created with
0.13.0-

SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"

   at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI
ndexShardGateway.java:

  1. at org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the
Lucene

indices to see if my analyzer configuration was doing the right
thing.

If I use ES 0.12.1 and then open the underlying Lucene index
with Luke

1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got
from git

yesterday) it is failing with an error "No valid directory at
the

location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.


(Sebastian Gavarini) #9

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://github.com/elasticsearch/elasticsearch/issues/closed/#issue/463)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:http://lucene.apache.org/java/3_0_2/fileformats.html#File
Naming http://lucene.apache.org/java/3_0_2/fileformats.html#File+Naming
http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt is
internal ES file (checksum file) and Lucene will not understand it. (Check
AbstractStore class in ES, method writeChecksum). Not sure why Lucene tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com wrote:

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at
    

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at
    

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older
ES

version (0.11.0 or something like that). The point at that time was
that I

was not able to open Lucene index from gateway in Luke but I was
able to

open the same "index" from work directory. Hope I remember correctly
:slight_smile:

But sure many things have changed since then. Local gateway was
added, index

is not compound now and in recent 0.13.0-SNAPSHOT version there are
some

breaking changes compared to 0.12.x version (checkhttps://
github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note

the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com
wrote:

In fact if I try to open the same data path index created with
0.13.0-

SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"

   at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI
ndexShardGateway.java:

  1. at org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the
Lucene

indices to see if my analyzer configuration was doing the right
thing.

If I use ES 0.12.1 and then open the underlying Lucene index
with Luke

1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got
from git

yesterday) it is failing with an error "No valid directory at
the

location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.


(Lukáš Vlček) #10

I would rather ask why Lucene does not ignore this checknum file when
opening the index. Should the checksum file had different naming convention
then Lucene would have been ignoring it?

On Tue, Nov 9, 2010 at 2:11 AM, Sebastian sgavarini@gmail.com wrote:

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://
github.com/elasticsearch/elasticsearch/issues/closed/#issue/463)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
http://lucene.apache.org/java/3_0_2/fileformats.html#File

Naming <
http://lucene.apache.org/java/3_0_2/fileformats.html#File+Naming>

http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt
is

internal ES file (checksum file) and Lucene will not understand it.
(Check

AbstractStore class in ES, method writeChecksum). Not sure why Lucene
tries

to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com wrote:

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene
IndexReader

and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException:
For

input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at
    

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at
    

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't
find

something that could explain Lucene index format
incompatibilities. I

am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway
and

work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some
older

ES

version (0.11.0 or something like that). The point at that time
was

that I

was not able to open Lucene index from gateway in Luke but I was
able to

open the same "index" from work directory. Hope I remember
correctly

:slight_smile:

But sure many things have changed since then. Local gateway was
added, index

is not compound now and in recent 0.13.0-SNAPSHOT version there
are

some

breaking changes compared to 0.12.x version (checkhttps://
github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note

the release notes for version under development may be delayed
or

incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com
wrote:

In fact if I try to open the same data path index created with
0.13.0-

SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService

$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"

   at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI

ndexShardGateway.java:

  1. at
    org.elasticsearch.index.gateway.IndexShardGatewayService

$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the
Lucene

indices to see if my analyzer configuration was doing the
right

thing.

If I use ES 0.12.1 and then open the underlying Lucene index
with Luke

1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got
from git

yesterday) it is failing with an error "No valid directory
at

the

location, try another location.".
Do you know what could be happening? any ideas how to fix
it?

Thanks,
Sebastian.


(Shay Banon) #11

Hi,

Yea, I see what happens, let me see if its something I can work around...

-shay.banon

On Tue, Nov 9, 2010 at 3:19 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

I would rather ask why Lucene does not ignore this checknum file when
opening the index. Should the checksum file had different naming convention
then Lucene would have been ignoring it?

On Tue, Nov 9, 2010 at 2:11 AM, Sebastian sgavarini@gmail.com wrote:

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://
github.com/elasticsearch/elasticsearch/issues/closed/#issue/463)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
http://lucene.apache.org/java/3_0_2/fileformats.html#File

Naming <
http://lucene.apache.org/java/3_0_2/fileformats.html#File+Naming>

http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt
is

internal ES file (checksum file) and Lucene will not understand it.
(Check

AbstractStore class in ES, method writeChecksum). Not sure why Lucene
tries

to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com
wrote:

Ok, the line causing the problem is:

line 199: return

Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),

Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene
IndexReader

and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException:
For

input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at
    

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at
    

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't
find

something that could explain Lucene index format
incompatibilities. I

am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway
and

work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some
older

ES

version (0.11.0 or something like that). The point at that time
was

that I

was not able to open Lucene index from gateway in Luke but I
was

able to

open the same "index" from work directory. Hope I remember
correctly

:slight_smile:

But sure many things have changed since then. Local gateway was
added, index

is not compound now and in recent 0.13.0-SNAPSHOT version there
are

some

breaking changes compared to 0.12.x version (checkhttps://
github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note

the release notes for version under development may be delayed
or

incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian <sgavar...@gmail.com

wrote:

In fact if I try to open the same data path index created
with

0.13.0-

SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService

$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"

   at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI

ndexShardGateway.java:

  1. at
    org.elasticsearch.index.gateway.IndexShardGatewayService

$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at
the

Lucene

indices to see if my analyzer configuration was doing the
right

thing.

If I use ES 0.12.1 and then open the underlying Lucene
index

with Luke

1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master
got

from git

yesterday) it is failing with an error "No valid directory
at

the

location, try another location.".
Do you know what could be happening? any ideas how to fix
it?

Thanks,
Sebastian.


(Sebastian Gavarini) #12

Hi Shay,

I think that being able to inspect the index with Luke is a nice
option, at least when checking your mappings are correct, or if
something is going wrong as a debug measure.

Now that I know that by (re)moving the ".cks" files it works, it's not
a problem for me, but probably many more developers are going to hit
the same thing when master gets released. Maybe moving the checksums
to another folder (/checksum? or whatever) so Lucene can read it's
index as it pleases, and that way also you guarantee future
compatibility even if Lucene chooses to add a ".cks" file extension to
it's list. I don't know, I think, it feels tidier to avoid Lucene's
index file structure.

Thanks,
Sebastian.

On Nov 9, 4:13 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Yea, I see what happens, let me see if its something I can work around...

-shay.banon

On Tue, Nov 9, 2010 at 3:19 AM, Lukáš Vlček lukas.vl...@gmail.com wrote:

I would rather ask why Lucene does not ignore this checknum file when
opening the index. Should the checksum file had different naming convention
then Lucene would have been ignoring it?

On Tue, Nov 9, 2010 at 2:11 AM, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://
github.com/elasticsearch/elasticsearch/issues/closed/#issue/463)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
http://lucene.apache.org/java/3_0_2/fileformats.html#File

Naming <
http://lucene.apache.org/java/3_0_2/fileformats.html#File+Naming>

http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt
is

internal ES file (checksum file) and Lucene will not understand it.
(Check

AbstractStore class in ES, method writeChecksum). Not sure why Lucene
tries

to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com
wrote:

Ok, the line causing the problem is:

line 199: return

Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),

Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene
IndexReader

and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException:
For

input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at
    

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at
    

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't
find

something that could explain Lucene index format
incompatibilities. I

am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway
and

work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some
older

ES

version (0.11.0 or something like that). The point at that time
was

that I

was not able to open Lucene index from gateway in Luke but I
was

able to

open the same "index" from work directory. Hope I remember
correctly

:slight_smile:

But sure many things have changed since then. Local gateway was
added, index

is not compound now and in recent 0.13.0-SNAPSHOT version there
are

some

breaking changes compared to 0.12.x version (checkhttps://
github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note

the release notes for version under development may be delayed
or

incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian <sgavar...@gmail.com

wrote:

In fact if I try to open the same data path index created
with

0.13.0-

SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService

$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"

   at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI

ndexShardGateway.java:

  1. at
    org.elasticsearch.index.gateway.IndexShardGatewayService

$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at
the

Lucene

indices to see if my analyzer configuration was doing the
right

thing.

If I use ES 0.12.1 and then open the underlying Lucene
index

with Luke

1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master
got

from git

yesterday) it is failing with an error "No valid directory
at

the

location, try another location.".
Do you know what could be happening? any ideas how to fix
it?

Thanks,
Sebastian.


(Shay Banon) #13

I understand, thats why I said I will try and work around it. Keeping it in
the same location is important to keep things in the same place.

On Tue, Nov 9, 2010 at 9:26 AM, Sebastian sgavarini@gmail.com wrote:

Hi Shay,

I think that being able to inspect the index with Luke is a nice
option, at least when checking your mappings are correct, or if
something is going wrong as a debug measure.

Now that I know that by (re)moving the ".cks" files it works, it's not
a problem for me, but probably many more developers are going to hit
the same thing when master gets released. Maybe moving the checksums
to another folder (/checksum? or whatever) so Lucene can read it's
index as it pleases, and that way also you guarantee future
compatibility even if Lucene chooses to add a ".cks" file extension to
it's list. I don't know, I think, it feels tidier to avoid Lucene's
index file structure.

Thanks,
Sebastian.

On Nov 9, 4:13 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Yea, I see what happens, let me see if its something I can work
around...

-shay.banon

On Tue, Nov 9, 2010 at 3:19 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I would rather ask why Lucene does not ignore this checknum file when
opening the index. Should the checksum file had different naming
convention

then Lucene would have been ignoring it?

On Tue, Nov 9, 2010 at 2:11 AM, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://
github.com/elasticsearch/elasticsearch/issues/closed/#issue/463)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
http://lucene.apache.org/java/3_0_2/fileformats.html#File

Naming <
http://lucene.apache.org/java/3_0_2/fileformats.html#File+Naming>

<
http://lucene.apache.org/java/3_0_2/fileformats.html#File+Naming>It

is

internal ES file (checksum file) and Lucene will not understand
it.

(Check

AbstractStore class in ES, method writeChecksum). Not sure why
Lucene

tries

to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com
wrote:

Ok, the line causing the problem is:

line 199: return

Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),

Character.MAX_RADIX);
in class SegmentInfos

the value of:
fileName.substring(1+IndexFileNames.SEGMENTS.length())

is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working
as

in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene
IndexReader

and I see the same error:

File file = new
File("/tmp/work/elasticsearch/test/nodes/0/indices/

tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread"
java.lang.NumberFormatException:

For

input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at
    

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at

org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:

  1. at
    

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but
didn't

find

something that could explain Lucene index format
incompatibilities. I

am using local gateway with default compound index format.

How was your previous problem solved? it is strange that
gateway

and

work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I remember I was getting similar error while I was using
some

older

ES

version (0.11.0 or something like that). The point at that
time

was

that I

was not able to open Lucene index from gateway in Luke but
I

was

able to

open the same "index" from work directory. Hope I remember
correctly

:slight_smile:

But sure many things have changed since then. Local gateway
was

added, index

is not compound now and in recent 0.13.0-SNAPSHOT version
there

are

some

breaking changes compared to 0.12.x version (checkhttps://
github.com/elasticsearch/elasticsearch/wiki/Release-notes, but
note

the release notes for version under development may be
delayed

or

incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian <
sgavar...@gmail.com

wrote:

In fact if I try to open the same data path index created
with

0.13.0-

SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster
]

[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService

$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input
string:

"2.cks"

   at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI

ndexShardGateway.java:

  1. at
    org.elasticsearch.index.gateway.IndexShardGatewayService

$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com
wrote:

Hi all,
Something weird is happening, I wanted to take a look
at

the

Lucene

indices to see if my analyzer configuration was doing
the

right

thing.

If I use ES 0.12.1 and then open the underlying Lucene
index

with Luke

1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT
(master

got

from git

yesterday) it is failing with an error "No valid
directory

at

the

location, try another location.".
Do you know what could be happening? any ideas how to
fix

it?

Thanks,
Sebastian.


(system) #14