Luke not recognizing index from ES git master

Elastic Stack Elasticsearch
1

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.

2

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at
org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(SegmentInfos.java:
199)
at
org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
134)
at
org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.

3

I remember I was getting similar error while I was using some older ES
version (0.11.0 or something like that). The point at that time was that I
was not able to open Lucene index from gateway in Luke but I was able to
open the same "index" from work directory. Hope I remember correctly :slight_smile:
But sure many things have changed since then. Local gateway was added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are some
breaking changes compared to 0.12.x version (check
GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavarini@gmail.com wrote:

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(SegmentInfos.java:
199)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
134)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.

4

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older ES
version (0.11.0 or something like that). The point at that time was that I
was not able to open Lucene index from gateway in Luke but I was able to
open the same "index" from work directory. Hope I remember correctly :slight_smile:
But sure many things have changed since then. Local gateway was added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are some
breaking changes compared to 0.12.x version (checkhttps://github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com wrote:

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment Infos.java:
199)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
134)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.

5

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at
org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(SegmentInfos.java:
199)
at
org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInfos.java:
134)
at org.apache.lucene.index.SegmentInfos
$FindSegmentsFile.run(SegmentInfos.java:571)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older ES
version (0.11.0 or something like that). The point at that time was that I
was not able to open Lucene index from gateway in Luke but I was able to
open the same "index" from work directory. Hope I remember correctly :slight_smile:
But sure many things have changed since then. Local gateway was added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are some
breaking changes compared to 0.12.x version (checkhttps://github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com wrote:

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment Infos.java:
199)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
134)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.

6

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at
org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment Infos.java:
199)
at
org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
134)
at org.apache.lucene.index.SegmentInfos
$FindSegmentsFile.run(SegmentInfos.java:571)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older ES
version (0.11.0 or something like that). The point at that time was that I
was not able to open Lucene index from gateway in Luke but I was able to
open the same "index" from work directory. Hope I remember correctly :slight_smile:
But sure many things have changed since then. Local gateway was added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are some
breaking changes compared to 0.12.x version (checkhttps://github.com/elasticsearch/elasticsearch/wiki/Release-notes, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com wrote:

In fact if I try to open the same data path index created with 0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string: "2.cks"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:
48)
at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment Infos.java:
199)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
134)
at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf os.java:
151)
at
org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)
at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:
87)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:147)
... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the Lucene
indices to see if my analyzer configuration was doing the right thing.
If I use ES 0.12.1 and then open the underlying Lucene index with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.

7

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
http://lucene.apache.org/java/3_0_2/fileformats.html#File
Naming
<Apache Lucene - Index File Formats Naming>It is
internal ES file (checksum file) and Lucene will not understand it. (Check
AbstractStore class in ES, method writeChecksum). Not sure why Lucene tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavarini@gmail.com wrote:

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older
ES
version (0.11.0 or something like that). The point at that time was
that I
was not able to open Lucene index from gateway in Luke but I was able
to
open the same "index" from work directory. Hope I remember correctly
:slight_smile:
But sure many things have changed since then. Local gateway was
added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are
some
breaking changes compared to 0.12.x version (checkhttps://
GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com
wrote:

In fact if I try to open the same data path index created with
0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI
ndexShardGateway.java:

  1. at org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the
Lucene
indices to see if my analyzer configuration was doing the right
thing.
If I use ES 0.12.1 and then open the underlying Lucene index with
Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got from
git
yesterday) it is failing with an error "No valid directory at the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.

8

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (see
Issues · elastic/elasticsearch · GitHub)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file: Apache Lucene - Index File Formats
Naming http://lucene.apache.org/java/3_0_2/fileformats.html#File+Naming
http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt is
internal ES file (checksum file) and Lucene will not understand it. (Check
AbstractStore class in ES, method writeChecksum). Not sure why Lucene tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavarini@gmail.com wrote:

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older
ES
version (0.11.0 or something like that). The point at that time was
that I
was not able to open Lucene index from gateway in Luke but I was
able to
open the same "index" from work directory. Hope I remember correctly
:slight_smile:
But sure many things have changed since then. Local gateway was
added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are
some
breaking changes compared to 0.12.x version (checkhttps://
GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com
wrote:

In fact if I try to open the same data path index created with
0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI
ndexShardGateway.java:

  1. at org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the
Lucene
indices to see if my analyzer configuration was doing the right
thing.
If I use ES 0.12.1 and then open the underlying Lucene index
with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got
from git
yesterday) it is failing with an error "No valid directory at
the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.

9

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://github.com/elasticsearch/elasticsearch/issues/closed/#issue/463)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:Apache Lucene - Index File Formats
Naming http://lucene.apache.org/java/3_0_2/fileformats.html#File+Naming
http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt is
internal ES file (checksum file) and Lucene will not understand it. (Check
AbstractStore class in ES, method writeChecksum). Not sure why Lucene tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com wrote:

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException: For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't find
something that could explain Lucene index format incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some older
ES
version (0.11.0 or something like that). The point at that time was
that I
was not able to open Lucene index from gateway in Luke but I was
able to
open the same "index" from work directory. Hope I remember correctly
:slight_smile:
But sure many things have changed since then. Local gateway was
added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there are
some
breaking changes compared to 0.12.x version (checkhttps://
GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine, but note
the release notes for version under development may be delayed or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com
wrote:

In fact if I try to open the same data path index created with
0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment
Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf
os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI
ndexShardGateway.java:

  1. at org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the
Lucene
indices to see if my analyzer configuration was doing the right
thing.
If I use ES 0.12.1 and then open the underlying Lucene index
with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got
from git
yesterday) it is failing with an error "No valid directory at
the
location, try another location.".
Do you know what could be happening? any ideas how to fix it?
Thanks,
Sebastian.

10

I would rather ask why Lucene does not ignore this checknum file when
opening the index. Should the checksum file had different naming convention
then Lucene would have been ignoring it?

On Tue, Nov 9, 2010 at 2:11 AM, Sebastian sgavarini@gmail.com wrote:

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://
Issues · elastic/elasticsearch · GitHub)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
Apache Lucene - Index File Formats
Naming <
Apache Lucene - Index File Formats>
http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt
is
internal ES file (checksum file) and Lucene will not understand it.
(Check
AbstractStore class in ES, method writeChecksum). Not sure why Lucene
tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com wrote:

Ok, the line causing the problem is:

line 199: return
Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),
Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene
IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException:
For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't
find
something that could explain Lucene index format
incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway
and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some
older
ES
version (0.11.0 or something like that). The point at that time
was
that I
was not able to open Lucene index from gateway in Luke but I was
able to
open the same "index" from work directory. Hope I remember
correctly
:slight_smile:
But sure many things have changed since then. Local gateway was
added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there
are
some
breaking changes compared to 0.12.x version (checkhttps://
GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine, but note
the release notes for version under development may be delayed
or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian sgavar...@gmail.com
wrote:

In fact if I try to open the same data path index created with
0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI

ndexShardGateway.java:

  1. at
    org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at the
Lucene
indices to see if my analyzer configuration was doing the
right
thing.
If I use ES 0.12.1 and then open the underlying Lucene index
with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master got
from git
yesterday) it is failing with an error "No valid directory
at
the
location, try another location.".
Do you know what could be happening? any ideas how to fix
it?
Thanks,
Sebastian.

11

Hi,

Yea, I see what happens, let me see if its something I can work around...

-shay.banon

On Tue, Nov 9, 2010 at 3:19 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

I would rather ask why Lucene does not ignore this checknum file when
opening the index. Should the checksum file had different naming convention
then Lucene would have been ignoring it?

On Tue, Nov 9, 2010 at 2:11 AM, Sebastian sgavarini@gmail.com wrote:

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://
Issues · elastic/elasticsearch · GitHub)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
Apache Lucene - Index File Formats
Naming <
Apache Lucene - Index File Formats>
http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt
is
internal ES file (checksum file) and Lucene will not understand it.
(Check
AbstractStore class in ES, method writeChecksum). Not sure why Lucene
tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com
wrote:

Ok, the line causing the problem is:

line 199: return

Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),

Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene
IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException:
For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't
find
something that could explain Lucene index format
incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway
and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some
older
ES
version (0.11.0 or something like that). The point at that time
was
that I
was not able to open Lucene index from gateway in Luke but I
was
able to
open the same "index" from work directory. Hope I remember
correctly
:slight_smile:
But sure many things have changed since then. Local gateway was
added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there
are
some
breaking changes compared to 0.12.x version (checkhttps://
GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine, but note
the release notes for version under development may be delayed
or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian <sgavar...@gmail.com

wrote:

In fact if I try to open the same data path index created
with
0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI

ndexShardGateway.java:

  1. at
    org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at
the
Lucene
indices to see if my analyzer configuration was doing the
right
thing.
If I use ES 0.12.1 and then open the underlying Lucene
index
with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master
got
from git
yesterday) it is failing with an error "No valid directory
at
the
location, try another location.".
Do you know what could be happening? any ideas how to fix
it?
Thanks,
Sebastian.

12

Hi Shay,

I think that being able to inspect the index with Luke is a nice
option, at least when checking your mappings are correct, or if
something is going wrong as a debug measure.

Now that I know that by (re)moving the ".cks" files it works, it's not
a problem for me, but probably many more developers are going to hit
the same thing when master gets released. Maybe moving the checksums
to another folder (/checksum? or whatever) so Lucene can read it's
index as it pleases, and that way also you guarantee future
compatibility even if Lucene chooses to add a ".cks" file extension to
it's list. I don't know, I think, it feels tidier to avoid Lucene's
index file structure.

Thanks,
Sebastian.

On Nov 9, 4:13 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Yea, I see what happens, let me see if its something I can work around...

-shay.banon

On Tue, Nov 9, 2010 at 3:19 AM, Lukáš Vlček lukas.vl...@gmail.com wrote:

I would rather ask why Lucene does not ignore this checknum file when
opening the index. Should the checksum file had different naming convention
then Lucene would have been ignoring it?

On Tue, Nov 9, 2010 at 2:11 AM, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://
Issues · elastic/elasticsearch · GitHub)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
Apache Lucene - Index File Formats
Naming <
Apache Lucene - Index File Formats>
http://lucene.apache.org/java/3_0_2/fileformats.html#File+NamingIt
is
internal ES file (checksum file) and Lucene will not understand it.
(Check
AbstractStore class in ES, method writeChecksum). Not sure why Lucene
tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com
wrote:

Ok, the line causing the problem is:

line 199: return

Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),

Character.MAX_RADIX);
in class SegmentInfos

the value of: fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene
IndexReader
and I see the same error:

File file = new File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread" java.lang.NumberFormatException:
For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at
    org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:
  2. at

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but didn't
find
something that could explain Lucene index format
incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that gateway
and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

I remember I was getting similar error while I was using some
older
ES
version (0.11.0 or something like that). The point at that time
was
that I
was not able to open Lucene index from gateway in Luke but I
was
able to
open the same "index" from work directory. Hope I remember
correctly
:slight_smile:
But sure many things have changed since then. Local gateway was
added, index
is not compound now and in recent 0.13.0-SNAPSHOT version there
are
some
breaking changes compared to 0.12.x version (checkhttps://
GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine, but note
the release notes for version under development may be delayed
or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian <sgavar...@gmail.com

wrote:

In fact if I try to open the same data path index created
with
0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster ]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input string:
"2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI

ndexShardGateway.java:

  1. at
    org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com wrote:

Hi all,
Something weird is happening, I wanted to take a look at
the
Lucene
indices to see if my analyzer configuration was doing the
right
thing.
If I use ES 0.12.1 and then open the underlying Lucene
index
with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT (master
got
from git
yesterday) it is failing with an error "No valid directory
at
the
location, try another location.".
Do you know what could be happening? any ideas how to fix
it?
Thanks,
Sebastian.

13

I understand, thats why I said I will try and work around it. Keeping it in
the same location is important to keep things in the same place.

On Tue, Nov 9, 2010 at 9:26 AM, Sebastian sgavarini@gmail.com wrote:

Hi Shay,

I think that being able to inspect the index with Luke is a nice
option, at least when checking your mappings are correct, or if
something is going wrong as a debug measure.

Now that I know that by (re)moving the ".cks" files it works, it's not
a problem for me, but probably many more developers are going to hit
the same thing when master gets released. Maybe moving the checksums
to another folder (/checksum? or whatever) so Lucene can read it's
index as it pleases, and that way also you guarantee future
compatibility even if Lucene chooses to add a ".cks" file extension to
it's list. I don't know, I think, it feels tidier to avoid Lucene's
index file structure.

Thanks,
Sebastian.

On Nov 9, 4:13 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Yea, I see what happens, let me see if its something I can work
around...

-shay.banon

On Tue, Nov 9, 2010 at 3:19 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I would rather ask why Lucene does not ignore this checknum file when
opening the index. Should the checksum file had different naming
convention
then Lucene would have been ignoring it?

On Tue, Nov 9, 2010 at 2:11 AM, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

You are right, about the compound format, it was my mistake when
posting, I am using the non-compound format.
About the .cks files, when I removed them Luke and Lucene both opened
the index correctly (of course ES won't probably like it), but I
wonder if there isn't a more compatible way of checksumming so Luke
still works, maybe to a different /checksum folder or something like
that.

Sebastian.

On Nov 8, 10:05 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

And BTW 0.13.0 should not be using compound format by default unless
explicitly specified (seehttps://
Issues · elastic/elasticsearch · GitHub)

On Tue, Nov 9, 2010 at 1:59 AM, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I am just speculating but I think .cks is not a Lucene 3.0.2 file:
Apache Lucene - Index File Formats
Naming <
Apache Lucene - Index File Formats>
<
Apache Lucene - Index File Formats>It
is
internal ES file (checksum file) and Lucene will not understand
it.
(Check
AbstractStore class in ES, method writeChecksum). Not sure why
Lucene
tries
to open it.

On Tue, Nov 9, 2010 at 1:45 AM, Sebastian sgavar...@gmail.com
wrote:

Ok, the line causing the problem is:

line 199: return

Long.parseLong(fileName.substring(1+IndexFileNames.SEGMENTS.length()),

Character.MAX_RADIX);
in class SegmentInfos

the value of:
fileName.substring(1+IndexFileNames.SEGMENTS.length())
is "2.cks", that's why the parseLong is failing.

What I would like to know is how can I fix it so it keeps working
as
in 0.12.1.

Thanks and sorry for all the mails.
Sebastian.

On Nov 8, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

I coded a small program that access the index using Lucene
IndexReader
and I see the same error:

File file = new
File("/tmp/work/elasticsearch/test/nodes/0/indices/
tmpindex/0/index");
Directory directory = new NIOFSDirectory(file);
IndexReader reader = IndexReader.open(directory);

Exception in thread "Main Thread"
java.lang.NumberFormatException:
For
input string: "2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at org.apache.lucene.index.SegmentInfos
    $FindSegmentsFile.run(SegmentInfos.java:571)
    at

org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:

  1. at

org.apache.lucene.index.IndexReader.open(IndexReader.java:316)

    at

org.apache.lucene.index.IndexReader.open(IndexReader.java:188)

Sebastian

On Nov 8, 9:32 pm, Sebastian sgavar...@gmail.com wrote:

Hi Lukas,

Thanks for your answer. I looked at the release notes but
didn't
find
something that could explain Lucene index format
incompatibilities. I
am using local gateway with default compound index format.

How was your previous problem solved? it is strange that
gateway
and
work were holding different representations.

Sebastian.

On Nov 8, 9:18 pm, Lukáš Vlček lukas.vl...@gmail.com
wrote:

I remember I was getting similar error while I was using
some
older
ES
version (0.11.0 or something like that). The point at that
time
was
that I
was not able to open Lucene index from gateway in Luke but
I
was
able to
open the same "index" from work directory. Hope I remember
correctly
:slight_smile:
But sure many things have changed since then. Local gateway
was
added, index
is not compound now and in recent 0.13.0-SNAPSHOT version
there
are
some
breaking changes compared to 0.12.x version (checkhttps://
GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine, but
note
the release notes for version under development may be
delayed
or
incomplete).

Lukas

On Tue, Nov 9, 2010 at 1:10 AM, Sebastian <
sgavar...@gmail.com

wrote:

In fact if I try to open the same data path index created
with
0.13.0-
SNAPSHOT with 0.12.1 I see:

[2010-11-08 21:08:30,258][WARN ][indices.cluster
]
[Nightshade] [test][0] failed to start shard

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[test][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:197)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NumberFormatException: For input
string:
"2.cks"
at

java.lang.NumberFormatException.forInputString(NumberFormatException.java:

  1. at java.lang.Long.parseLong(Long.java:412)
    at

org.apache.lucene.index.SegmentInfos.generationFromSegmentsFileName(Segment

Infos.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.SegmentInfos.getCurrentSegmentGeneration(SegmentInf

os.java:

  1. at

org.apache.lucene.index.IndexReader.indexExists(IndexReader.java:626)

   at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI

ndexShardGateway.java:

  1. at
    org.elasticsearch.index.gateway.IndexShardGatewayService
    $1.run(IndexShardGatewayService.java:147)
    ... 3 more

Had something changed regarding lucene version?

On Nov 8, 8:39 pm, Sebastian sgavar...@gmail.com
wrote:

Hi all,
Something weird is happening, I wanted to take a look
at
the
Lucene
indices to see if my analyzer configuration was doing
the
right
thing.
If I use ES 0.12.1 and then open the underlying Lucene
index
with Luke
1.0.1 it works fine, but with ES 0.13.0-SNAPSHOT
(master
got
from git
yesterday) it is failing with an error "No valid
directory
at
the
location, try another location.".
Do you know what could be happening? any ideas how to
fix
it?
Thanks,
Sebastian.