0.90.0 Upgrade

Hi everyone,

I've tried the 0.90.0 upgrade from a 0.19.9 containing 15 indices. I've
kept data/, replaced bin/, lib/,config/ and cleaned logs/
When I now start ES, it tells me "recovered [6] indices into
cluster_state". More than half of of my indices are not visible and there's
no way to request on them.

I've then tried an upgrade from 0.19.9 to 0.20.6 (using a backup of my
data/) and I had lots of errors of this kind :
[2013-05-14 14:41:20,630][WARN ][cluster.action.shard ] [Silver
Scorpion] received shard failed for [rss][2], node[byDaPG55Q2SotRaUUV70Pg],
[P], s[INITIALIZING],
reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[rss][2] shard allocated for local
recovery (post api), should exists, but doesn't]]]

Those shards were present in the 0.19.9 version and there was no error.
Despite this, indices are visible though.
Then I upgraded from 0.20.6 to 0.90.0 and ES found three more indices
(that's 9). Unfortunately not all of them.

Anybody ever had this issue? Is it possible to upgrade from 0.19.xx to
0.90.0?
Thanks in advance !

Regards,
Sébastien

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I tried it in debug mode. I have this kind of exception each time one of
the index is discarded (I think):

org.elasticsearch.common.jackson.core.JsonParseException: Unexpected
character ('s' (code 115)): expected a valid value (number, String, array,
object, 'true', 'false' or 'null')
at [Source: [B@7eb6a094; line: 1, column: 2]
at
org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1378)
at
org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:599)
at
org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:520)
at
org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2275)
at
org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:788)
at
org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:674)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:50)
at
org.elasticsearch.common.xcontent.support.XContentMapConverter.readMap(XContentMapConverter.java:67)
at
org.elasticsearch.common.xcontent.support.XContentMapConverter.readOrderedMap(XContentMapConverter.java:60)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrdered(AbstractXContentParser.java:126)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrderedAndClose(AbstractXContentParser.java:141)
at
org.elasticsearch.cluster.metadata.MappingMetaData.(MappingMetaData.java:272)
at
org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:613)
at
org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.loadIndex(LocalGatewayMetaState.java:494)
at
org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.clusterChanged(LocalGatewayMetaState.java:251)
at
org.elasticsearch.gateway.local.LocalGateway.clusterChanged(LocalGateway.java:206)
at
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:324)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:95)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

A mapping issue?

Le mardi 14 mai 2013 17:57:12 UTC+2, Sébastien Sabat a écrit :

Hi everyone,

I've tried the 0.90.0 upgrade from a 0.19.9 containing 15 indices. I've
kept data/, replaced bin/, lib/,config/ and cleaned logs/
When I now start ES, it tells me "recovered [6] indices into
cluster_state". More than half of of my indices are not visible and there's
no way to request on them.

I've then tried an upgrade from 0.19.9 to 0.20.6 (using a backup of my
data/) and I had lots of errors of this kind :
[2013-05-14 14:41:20,630][WARN ][cluster.action.shard ] [Silver
Scorpion] received shard failed for [rss][2], node[byDaPG55Q2SotRaUUV70Pg],
[P], s[INITIALIZING],
reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[rss][2] shard allocated for local
recovery (post api), should exists, but doesn't]]]

Those shards were present in the 0.19.9 version and there was no error.
Despite this, indices are visible though.
Then I upgraded from 0.20.6 to 0.90.0 and ES found three more indices
(that's 9). Unfortunately not all of them.

Anybody ever had this issue? Is it possible to upgrade from 0.19.xx to
0.90.0?
Thanks in advance !

Regards,
Sébastien

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Sebastien

I would try starting up 0.19 again (on a backed up data dir), then:

  • disable flush
  • stop indexing
  • issue a manual flush (to clear out the transaction log)
  • then shutdown and upgrade to 0.90

It looks to me like the transaction log is screwy

clint

On Wed, May 15, 2013 at 11:39 AM, Sébastien Sabat ssabat2000@gmail.comwrote:

I tried it in debug mode. I have this kind of exception each time one of
the index is discarded (I think):

org.elasticsearch.common.**jackson.core.**JsonParseException: Unexpected
character ('s' (code 115)): expected a valid value (number, String, array,
object, 'true', 'false' or 'null')
at [Source: [B@7eb6a094; line: 1, column: 2]
at org.elasticsearch.common.jackson.core.JsonParser._
constructError(JsonParser.**java:1378)
at org.elasticsearch.common.jackson.core.base.
ParserMinimalBase.**reportError(ParserMinimalBase.**java:599)
at org.elasticsearch.common.jackson.core.base.
ParserMinimalBase.
**reportUnexpectedChar(**ParserMinimalBase.java:520)
at org.elasticsearch.common.jackson.core.json.
UTF8StreamJsonParser.**handleUnexpectedValue(**UTF8StreamJsonParser.java:
**2275)
at org.elasticsearch.common.jackson.core.json.
UTF8StreamJsonParser.
nextTokenNotInObject(
UTF8StreamJsonParser.java:788)
at org.elasticsearch.common.jackson.core.json.
UTF8StreamJsonParser.**nextToken(**UTF8StreamJsonParser.java:674)
at org.elasticsearch.common.xcontent.json.
JsonXContentParser.nextToken(**JsonXContentParser.java:50)
at org.elasticsearch.common.xcontent.support.
XContentMapConverter.readMap(**XContentMapConverter.java:67)
at org.elasticsearch.common.xcontent.support.
XContentMapConverter.**readOrderedMap(**XContentMapConverter.java:60)
at org.elasticsearch.common.xcontent.support.
AbstractXContentParser.**mapOrdered(**AbstractXContentParser.java:**126)
at org.elasticsearch.common.xcontent.support.
AbstractXContentParser.**mapOrderedAndClose(**AbstractXContentParser.java:
**141)
at org.elasticsearch.cluster.metadata.MappingMetaData.<
init>(MappingMetaData.java:**272)
at org.elasticsearch.cluster.metadata.IndexMetaData$
Builder.fromXContent(**IndexMetaData.java:613)
at org.elasticsearch.gateway.local.state.meta.
LocalGatewayMetaState.**loadIndex(**LocalGatewayMetaState.java:**494)
at org.elasticsearch.gateway.local.state.meta.
LocalGatewayMetaState.**clusterChanged(**LocalGatewayMetaState.java:**251)
at org.elasticsearch.gateway.local.LocalGateway.
clusterChanged(LocalGateway.*java:206)
at org.elasticsearch.cluster.service.
InternalClusterService$2.run(InternalClusterService.java:324)
at org.elasticsearch.common.util.concurrent.
PrioritizedEsThreadPoolExecuto
r$TieBreakingPrioritizedRunnable.run(

*PrioritizedEsThreadPoolExecuto
r.java:95)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.**java:722)

A mapping issue?

Le mardi 14 mai 2013 17:57:12 UTC+2, Sébastien Sabat a écrit :

Hi everyone,

I've tried the 0.90.0 upgrade from a 0.19.9 containing 15 indices. I've
kept data/, replaced bin/, lib/,config/ and cleaned logs/
When I now start ES, it tells me "recovered [6] indices into
cluster_state". More than half of of my indices are not visible and there's
no way to request on them.

I've then tried an upgrade from 0.19.9 to 0.20.6 (using a backup of my
data/) and I had lots of errors of this kind :
[2013-05-14 14:41:20,630][WARN ][cluster.action.shard ] [Silver
Scorpion] received shard failed for [rss][2], node[byDaPG55Q2SotRaUUV70Pg],
[P], s[INITIALIZING],
reason [Failed to start shard, message [**IndexShardGatewayRecoveryExcep
**tion[[rss][2] shard allocated for local recovery (post api), should
exists, but doesn't]]]

Those shards were present in the 0.19.9 version and there was no error.
Despite this, indices are visible though.
Then I upgraded from 0.20.6 to 0.90.0 and ES found three more indices
(that's 9). Unfortunately not all of them.

Anybody ever had this issue? Is it possible to upgrade from 0.19.xx to
0.90.0?
Thanks in advance !

Regards,
Sébastien

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Clinton,

Thanks for your help.
I 've tried with another base in 0.19.9 and upgraded it to 0.26.0.
Everything was fine.
I used a manual flush and made sure there were no indexation.
Then I upgraded to 0.90.0 and the problem is still here : more than half my
indices are not visible.

But I launched ES in debug mode and saw this :

[2013-05-14 19:55:11,069][DEBUG][gateway.local.state.meta ] [Apache Kid]
[indexXXX]: failed to read
[C:\Dev\Elasticsearch\elasticsearch\data\elasticsearch\nodes\0\indices\indexXXX_state\state-10],
ignoring...
[2013-05-14 19:55:08,841][DEBUG][gateway.local.state.meta ] [Apache Kid]
[indexXXX] failed to find metadata for existing index location

It's like ES cannot read some metadata files. And then skip the index.
There's also a recurrent exception :

org.elasticsearch.common.jackson.core.JsonParseException: Unexpected
character ('s' (code 115)): expected a valid value (number, String, array,
object, 'true', 'false' or 'null')
at [Source: [B@7eb6a094; line: 1, column: 2]
at
org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1378)
at
org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:599)
at
org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:520)
at
org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2275)
at
org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:788)
at
org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:674)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:50)
at
org.elasticsearch.common.xcontent.support.XContentMapConverter.readMap(XContentMapConverter.java:67)
at
org.elasticsearch.common.xcontent.support.XContentMapConverter.readOrderedMap(XContentMapConverter.java:60)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrdered(AbstractXContentParser.java:126)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrderedAndClose(AbstractXContentParser.java:141)
at
org.elasticsearch.cluster.metadata.MappingMetaData.(MappingMetaData.java:272)
at
org.elasticsearch.cluster.metadata.IndexMetaData$Builder.fromXContent(IndexMetaData.java:613)
at
org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.loadIndex(LocalGatewayMetaState.java:494)
at
org.elasticsearch.gateway.local.state.meta.LocalGatewayMetaState.clusterChanged(LocalGatewayMetaState.java:251)
at
org.elasticsearch.gateway.local.LocalGateway.clusterChanged(LocalGateway.java:206)
at
org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:324)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:95)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

Le mercredi 15 mai 2013 11:41:35 UTC+2, Clinton Gormley a écrit :

Hi Sebastien

I would try starting up 0.19 again (on a backed up data dir), then:

  • disable flush
  • stop indexing
  • issue a manual flush (to clear out the transaction log)
  • then shutdown and upgrade to 0.90

It looks to me like the transaction log is screwy

clint

On Wed, May 15, 2013 at 11:39 AM, Sébastien Sabat <ssaba...@gmail.com<javascript:>

wrote:

I tried it in debug mode. I have this kind of exception each time one of
the index is discarded (I think):

org.elasticsearch.common.**jackson.core.**JsonParseException: Unexpected
character ('s' (code 115)): expected a valid value (number, String, array,
object, 'true', 'false' or 'null')
at [Source: [B@7eb6a094; line: 1, column: 2]
at org.elasticsearch.common.jackson.core.JsonParser._
constructError(JsonParser.**java:1378)
at org.elasticsearch.common.jackson.core.base.
ParserMinimalBase.**reportError(ParserMinimalBase.**java:599)
at org.elasticsearch.common.jackson.core.base.
ParserMinimalBase.
**reportUnexpectedChar(**ParserMinimalBase.java:520)
at org.elasticsearch.common.jackson.core.json.
UTF8StreamJsonParser.handleUnexpectedValue(
UTF8StreamJsonParser.java:**2275)
at org.elasticsearch.common.jackson.core.json.
UTF8StreamJsonParser.
nextTokenNotInObject(
UTF8StreamJsonParser.java:788)
at org.elasticsearch.common.jackson.core.json.
UTF8StreamJsonParser.**nextToken(**UTF8StreamJsonParser.java:674)
at org.elasticsearch.common.xcontent.json.
JsonXContentParser.nextToken(**JsonXContentParser.java:50)
at org.elasticsearch.common.xcontent.support.
XContentMapConverter.readMap(**XContentMapConverter.java:67)
at org.elasticsearch.common.xcontent.support.
XContentMapConverter.**readOrderedMap(**XContentMapConverter.java:60)
at org.elasticsearch.common.xcontent.support.
AbstractXContentParser.**mapOrdered(**AbstractXContentParser.java:**126)
at org.elasticsearch.common.xcontent.support.
AbstractXContentParser.mapOrderedAndClose(
AbstractXContentParser.java:**141)
at org.elasticsearch.cluster.metadata.MappingMetaData.<
init>(MappingMetaData.java:**272)
at org.elasticsearch.cluster.metadata.IndexMetaData$
Builder.fromXContent(**IndexMetaData.java:613)
at org.elasticsearch.gateway.local.state.meta.
LocalGatewayMetaState.**loadIndex(**LocalGatewayMetaState.java:**494)
at org.elasticsearch.gateway.local.state.meta.
LocalGatewayMetaState.**clusterChanged(LocalGatewayMetaState.java:
251)
at org.elasticsearch.gateway.local.LocalGateway.
clusterChanged(LocalGateway.**java:206)
at org.elasticsearch.cluster.service.
InternalClusterService$2.run(**InternalClusterService.java:324)
at org.elasticsearch.common.util.concurrent.
PrioritizedEsThreadPoolExecuto
r$TieBreakingPrioritizedRunnable.run(
PrioritizedEsThreadPoolExecutor.java:95)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.**java:722)

A mapping issue?

Le mardi 14 mai 2013 17:57:12 UTC+2, Sébastien Sabat a écrit :

Hi everyone,

I've tried the 0.90.0 upgrade from a 0.19.9 containing 15 indices. I've
kept data/, replaced bin/, lib/,config/ and cleaned logs/
When I now start ES, it tells me "recovered [6] indices into
cluster_state". More than half of of my indices are not visible and there's
no way to request on them.

I've then tried an upgrade from 0.19.9 to 0.20.6 (using a backup of my
data/) and I had lots of errors of this kind :
[2013-05-14 14:41:20,630][WARN ][cluster.action.shard ] [Silver
Scorpion] received shard failed for [rss][2], node[byDaPG55Q2SotRaUUV70Pg],
[P], s[INITIALIZING],
reason [Failed to start shard, message [**
IndexShardGatewayRecoveryExcep**tion[[rss][2] shard allocated for local
recovery (post api), should exists, but doesn't]]]

Those shards were present in the 0.19.9 version and there was no error.
Despite this, indices are visible though.
Then I upgraded from 0.20.6 to 0.90.0 and ES found three more indices
(that's 9). Unfortunately not all of them.

Anybody ever had this issue? Is it possible to upgrade from 0.19.xx to
0.90.0?
Thanks in advance !

Regards,
Sébastien

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.