Can't open file to read checksums

Elastic Stack Elasticsearch
1

Hi!

We get the following errors, on two of our nodes. And after that our
cluster doesn't work. I have no idea what it means.

[2014-08-25 17:46:39,323][WARN ][indices.store ]
[p-elasticlog03] Can't open file to read checksums
java.io.FileNotFoundException: No such file [_6cq_es090_0.doc]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:173)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:532)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:459)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
at
org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:271)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:186)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:140)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:277)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:268)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Br
Casper

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5f417878-3e49-478d-90e7-ca5c42734567%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

A few questions:

What version of Elasticsearch are you using?
Are you using the Java client and is it the same version of the cluster?
Did you upgrade recently and was the index built with an older version of
Elasticsearch?

Elasticsearch recently added checksum verification (1.3?), so perhaps you
have some sort of version mismatch.

Cheers,

Ivan

On Mon, Aug 25, 2014 at 10:52 AM, Casper Thrane casper.s.thrane@gmail.com
wrote:

Hi!

We get the following errors, on two of our nodes. And after that our
cluster doesn't work. I have no idea what it means.

[2014-08-25 17:46:39,323][WARN ][indices.store ]
[p-elasticlog03] Can't open file to read checksums
java.io.FileNotFoundException: No such file [_6cq_es090_0.doc]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:173)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:532)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:459)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
at
org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:271)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:186)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:140)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:277)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:268)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Br
Casper

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5f417878-3e49-478d-90e7-ca5c42734567%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5f417878-3e49-478d-90e7-ca5c42734567%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBYGbr-k%2BB41UUwUY7DVk27KUiuf1xY5kekkXoNRc3grg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Getting a similar error after upgrading to ES 1.3.2 from 1.1.1, however my
cluster is still ingesting logs without issues:

2014-08-30 06:46:20,031][WARN ][index.store ] [Jocasta]
[logstash-2014.08.25][2] Can't open file to read checksums
java.io.FileNotFoundException: No such file [_13sb_es090_0.tim]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:173)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:461)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
at org.elasticsearch.index.store.Store.getMetadata(Store.java:144)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:728)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:580)
at
org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:184)
at
org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:444)
at
org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)

On Tuesday, August 26, 2014 1:40:24 PM UTC-7, Ivan Brusic wrote:

A few questions:

What version of Elasticsearch are you using?
Are you using the Java client and is it the same version of the cluster?
Did you upgrade recently and was the index built with an older version of
Elasticsearch?

Elasticsearch recently added checksum verification (1.3?), so perhaps you
have some sort of version mismatch.

Cheers,

Ivan

On Mon, Aug 25, 2014 at 10:52 AM, Casper Thrane <casper....@gmail.com
<javascript:>> wrote:

Hi!

We get the following errors, on two of our nodes. And after that our
cluster doesn't work. I have no idea what it means.

[2014-08-25 17:46:39,323][WARN ][indices.store ]
[p-elasticlog03] Can't open file to read checksums
java.io.FileNotFoundException: No such file [_6cq_es090_0.doc]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:173)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:532)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:459)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
at
org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:271)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:186)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:140)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:277)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:268)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Br
Casper

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5f417878-3e49-478d-90e7-ca5c42734567%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5f417878-3e49-478d-90e7-ca5c42734567%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2ba2d4c5-a529-4261-9330-d50c7ef1acb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Could it be a bug?
I saw the exactly same warning from log file while rolling restart all
nodes. I am running 1.3.2 on all nodes.
I saw about 4 times when restart total 12 nodes.
The cluster is OK. all indices eventually in green state.
One more detail, the warning happens on the node different from the node I
am restarting.

On Tuesday, August 26, 2014 1:40:24 PM UTC-7, Ivan Brusic wrote:

A few questions:

What version of Elasticsearch are you using?
Are you using the Java client and is it the same version of the cluster?
Did you upgrade recently and was the index built with an older version of
Elasticsearch?

Elasticsearch recently added checksum verification (1.3?), so perhaps you
have some sort of version mismatch.

Cheers,

Ivan

On Mon, Aug 25, 2014 at 10:52 AM, Casper Thrane <casper....@gmail.com
<javascript:>> wrote:

Hi!

We get the following errors, on two of our nodes. And after that our
cluster doesn't work. I have no idea what it means.

[2014-08-25 17:46:39,323][WARN ][indices.store ]
[p-elasticlog03] Can't open file to read checksums
java.io.FileNotFoundException: No such file [_6cq_es090_0.doc]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:173)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:532)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.buildMetadata(Store.java:459)
at
org.elasticsearch.index.store.Store$MetadataSnapshot.(Store.java:433)
at
org.elasticsearch.index.store.Store.readMetadataSnapshot(Store.java:271)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:186)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:140)
at
org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:277)
at
org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:268)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Br
Casper

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5f417878-3e49-478d-90e7-ca5c42734567%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5f417878-3e49-478d-90e7-ca5c42734567%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/38bcc040-bb59-45a4-babc-c5c4dadcb66e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.