No commit point data in gateway

James_Cook · May 20, 2011, 4:49pm

I'm seeing this error message appear in my startup logs, which seems to be
preventing the ES cluster from reaching YELLOW/GREEN state.

16:42:27,642 WARN thread-1 arch.indices.cluster: 90 - [Supernalia]
[ppkc][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[ppkc][1] No commit point data is available in gateway
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:423)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:144)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

In this gist is the full log output along with my ES configuration params.
Running 0.16.0 on EC2, clustering not working yet.

gist.github.com

https://gist.github.com/oravecz/983300

gistfile1.txt

16:42:02,832  INFO     main ropertiesFactoryBean: 177 - Loading properties file from class path resource [elasticsearch-server.properties]
16:42:02,843  INFO     main .ElasticSearchServer:  47 - Starting the Elastic Search server node
16:42:02,934  INFO     main g.elasticsearch.node:  78 - [Supernalia] {elasticsearch/0.16.0}[26109]: initializing ...
16:42:02,946  INFO     main lasticsearch.plugins:  78 - [Supernalia] loaded [cloud-aws]
16:42:05,329 DEBUG     main csearch.cache.memory:  70 - [Supernalia] using bytebuffer cache with small_buffer_size [1kb], large_buffer_size [1mb], small_cache_size [10mb], large_cache_size [500mb], direct [true]
16:42:05,440 DEBUG     main ticsearch.threadpool:  70 - [Supernalia] creating thread_pool [cached], type [cached], keep_alive [30s]
16:42:05,444 DEBUG     main ticsearch.threadpool:  70 - [Supernalia] creating thread_pool [index], type [cached], keep_alive [5m]
16:42:05,445 DEBUG     main ticsearch.threadpool:  70 - [Supernalia] creating thread_pool [search], type [cached], keep_alive [5m]
16:42:05,446 DEBUG     main ticsearch.threadpool:  70 - [Supernalia] creating thread_pool [percolate], type [cached], keep_alive [5m]
16:42:05,447 DEBUG     main ticsearch.threadpool:  70 - [Supernalia] creating thread_pool [management], type [scaling], min [1], size [20], keep_alive [30s]

This file has been truncated. show original

All I have done is redeploy a new version of my webapp to a running app
server instance. The EBS block is the same as the previous and ES is using
local indicies. I'm not sure what happended to the so-called commit point
data, but I have seen this error frequently.

kimchy · May 20, 2011, 10:08pm

The way that the s3 gateway works is that it writes a commit point file listing all the files relevant for that shard after it has written all the files. And then, when recovering, it reads the commit point file, and recover all the data. It seems like some files are missing for that shard at s3. Not really sure how it got to that state, but it means that the relevant shard data is basically lost.

Is this something that you can recreate consistently? I would like to try and recreate it.
On Friday, May 20, 2011 at 7:49 PM, James Cook wrote:

I'm seeing this error message appear in my startup logs, which seems to be preventing the ES cluster from reaching YELLOW/GREEN state.

16:42:27,642 WARN thread-1 arch.indices.cluster: 90 - [Supernalia] [ppkc][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [ppkc][1] No commit point data is available in gateway
at org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:423)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:144)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

In this gist is the full log output along with my ES configuration params. Running 0.16.0 on EC2, clustering not working yet.
gist:983300 · GitHub

All I have done is redeploy a new version of my webapp to a running app server instance. The EBS block is the same as the previous and ES is using local indicies. I'm not sure what happended to the so-called commit point data, but I have seen this error frequently.