ES index gateway issues


(cwho) #1

Hi,

I am running 0.16.2, although we've encountered the same issue with 0.16.4,
0.16.5.

Occasionally ES will, on startup, fail and show the error message
(repeatedly) below. It says that "No commit point data is available in
gateway". The current fix is to delete the gateway and index, and restart ES
and start over.

I'm running on a single machine, single server node (and one client node),
and using the gateway just to persist my index. My config file is also
below. Can you let me know what I might be doing wrong? The error occurences
are random, as far as I can tell....

Config File:

name: mycomp
cluster:
name: MyCluster
http:
enabled: true
port: 9200-9300
node:
data: true
path:
data: c:/Apps/elasticsearch-0.16.1/data
logs: c:/Apps/elasticsearch-0.16.1/logs

gateway:
type: fs
fs:
snapshot_interval: 10s
location: /home/escluster/store
index:
analysis:
analyzer:
whitespace:
type: custom
tokenizer: whitespace
filter: lowercase

network:
host: 127.0.0.1

Error message:

[2011-08-01 16:53:52,016][WARN ][cluster.action.shard ] [mycomp]sending
failed shard for [myindex][0], node[rH3S3DYATIKZr4ZJuOjfhQ], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[myindex][0] No commit point data is
available in gateway]]]
[2011-08-01 16:53:52,017][WARN ][cluster.action.shard ] [mycomp]received
shard failed for [myindex][0], node[rH3S3DYATIKZr4ZJuOjfhQ], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[myindex][0] No commit point data is
available in gateway]]]
[2011-08-01 16:53:52,102][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-o]/[24], but not all files
exists, ignoring
[2011-08-01 16:53:52,105][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-n]/[23], but not all files
exists, ignoring
[2011-08-01 16:53:52,106][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-m]/[22], but not all files
exists, ignoring
[2011-08-01 16:53:52,107][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-l]/[21], but not all files
exists, ignoring
[2011-08-01 16:53:52,108][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-k]/[20], but not all files
exists, ignoring
[2011-08-01 16:53:52,109][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-j]/[19], but not all files
exists, ignoring
[2011-08-01 16:53:52,110][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-i]/[18], but not all files
exists, ignoring
[2011-08-01 16:53:52,110][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-h]/[17], but not all files
exists, ignoring
[2011-08-01 16:53:52,111][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-g]/[16], but not all files
exists, ignoring
[2011-08-01 16:53:52,112][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-f]/[15], but not all files
exists, ignoring
[2011-08-01 16:53:52,113][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-e]/[14], but not all files
exists, ignoring
[2011-08-01 16:53:52,114][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-d]/[13], but not all files
exists, ignoring
[2011-08-01 16:53:52,115][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-c]/[12], but not all files
exists, ignoring
[2011-08-01 16:53:52,116][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-b]/[11], but not all files
exists, ignoring
[2011-08-01 16:53:52,117][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-a]/[10], but not all files
exists, ignoring
[2011-08-01 16:53:52,118][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-6]/[6], but not all files
exists, ignoring
[2011-08-01 16:53:52,119][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-5]/[5], but not all files
exists, ignoring
[2011-08-01 16:53:52,120][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-4]/[4], but not all files
exists, ignoring
[2011-08-01 16:53:52,120][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-3]/[3], but not all files
exists, ignoring
[2011-08-01 16:53:52,121][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-2]/[2], but not all files
exists, ignoring
[2011-08-01 16:53:52,122][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-1]/[1], but not all files
exists, ignoring
[2011-08-01 16:53:52,123][WARN ][indices.cluster ]
[mycomp][myindex][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[myindex][0] No commit point data is available in gateway
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:423)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:144)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)


(Shay Banon) #2

Seems like the gateway is missing some files. Is this something you can
recreate easily so I can try and recreate it?

A note on your configuration, I suggest moving to local gateway. It will
reduce the load on the system, and won't require snapshotting to another
directory (it can recover from its existing node data location), especially
in your case where you have a one node cluster. I assume you have been using
the shared gateway since before the local gateway was introduced, and never
changed it?

On Mon, Aug 1, 2011 at 1:13 PM, cwho80 fuzzybr80@gmail.com wrote:

Hi,

I am running 0.16.2, although we've encountered the same issue with 0.16.4,
0.16.5.

Occasionally ES will, on startup, fail and show the error message
(repeatedly) below. It says that "No commit point data is available in
gateway". The current fix is to delete the gateway and index, and restart ES
and start over.

I'm running on a single machine, single server node (and one client node),
and using the gateway just to persist my index. My config file is also
below. Can you let me know what I might be doing wrong? The error occurences
are random, as far as I can tell....

Config File:

name: mycomp
cluster:
name: MyCluster
http:
enabled: true
port: 9200-9300
node:
data: true
path:
data: c:/Apps/elasticsearch-0.16.1/data
logs: c:/Apps/elasticsearch-0.16.1/logs

gateway:
type: fs
fs:
snapshot_interval: 10s
location: /home/escluster/store
index:
analysis:
analyzer:
whitespace:
type: custom
tokenizer: whitespace
filter: lowercase

network:
host: 127.0.0.1

Error message:

[2011-08-01 16:53:52,016][WARN ][cluster.action.shard ] [mycomp]sending
failed shard for [myindex][0], node[rH3S3DYATIKZr4ZJuOjfhQ], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[myindex][0] No commit point data is
available in gateway]]]
[2011-08-01 16:53:52,017][WARN ][cluster.action.shard ]
[mycomp]received shard failed for [myindex][0],
node[rH3S3DYATIKZr4ZJuOjfhQ], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[myindex][0] No commit
point data is available in gateway]]]
[2011-08-01 16:53:52,102][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-o]/[24], but not all files
exists, ignoring
[2011-08-01 16:53:52,105][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-n]/[23], but not all files
exists, ignoring
[2011-08-01 16:53:52,106][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-m]/[22], but not all files
exists, ignoring
[2011-08-01 16:53:52,107][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-l]/[21], but not all files
exists, ignoring
[2011-08-01 16:53:52,108][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-k]/[20], but not all files
exists, ignoring
[2011-08-01 16:53:52,109][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-j]/[19], but not all files
exists, ignoring
[2011-08-01 16:53:52,110][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-i]/[18], but not all files
exists, ignoring
[2011-08-01 16:53:52,110][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-h]/[17], but not all files
exists, ignoring
[2011-08-01 16:53:52,111][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-g]/[16], but not all files
exists, ignoring
[2011-08-01 16:53:52,112][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-f]/[15], but not all files
exists, ignoring
[2011-08-01 16:53:52,113][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-e]/[14], but not all files
exists, ignoring
[2011-08-01 16:53:52,114][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-d]/[13], but not all files
exists, ignoring
[2011-08-01 16:53:52,115][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-c]/[12], but not all files
exists, ignoring
[2011-08-01 16:53:52,116][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-b]/[11], but not all files
exists, ignoring
[2011-08-01 16:53:52,117][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-a]/[10], but not all files
exists, ignoring
[2011-08-01 16:53:52,118][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-6]/[6], but not all files
exists, ignoring
[2011-08-01 16:53:52,119][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-5]/[5], but not all files
exists, ignoring
[2011-08-01 16:53:52,120][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-4]/[4], but not all files
exists, ignoring
[2011-08-01 16:53:52,120][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-3]/[3], but not all files
exists, ignoring
[2011-08-01 16:53:52,121][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-2]/[2], but not all files
exists, ignoring
[2011-08-01 16:53:52,122][WARN ][index.gateway.fs ]
[mycomp][myindex][0] listed commit_point [commit-1]/[1], but not all files
exists, ignoring
[2011-08-01 16:53:52,123][WARN ][indices.cluster ]
[mycomp][myindex][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[myindex][0] No commit point data is available in gateway
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:423)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:144)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)


(system) #3