Message not fully read and TransportShardReplicationOperationAction

Hello everybody,

Since several days i have regularly this message :

[2013-02-27 17:26:42,001][WARN ][transport.netty ] [node1] Message
not fully read (response) for [19644702] handler
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@111480a2,
error [true], resetting

There is apparently no problem because the cluster is usable.
But since the 'Message not fully read' message appeared, i got 2 times this
message (on different index) :

[2013-02-27 17:37:33,647][WARN ][index.merge.scheduler ] [node1]
[index_name][0] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (6 <= 6 )
(out: org.apache.lucene.store.FSDirectory$FSIndexOutput@4d02bc28)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4263)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

I have a 2 nodes ES cluster, each one is on 0.19.11 version.
Thank you for any piece of advice.
Germain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Finally, we had to upgrade the ES cluster from 0.19.11 to 0.20.5, and we
still have the same warning.

[2013-03-05 13:06:21,531][WARN ][transport.netty ] [node1] Message
not fully read (response) for [2049604] handler
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@7e41346c,
error [true], resetting

Le mercredi 27 février 2013 19:34:46 UTC+1, Gmaurice a écrit :

Hello everybody,

Since several days i have regularly this message :

[2013-02-27 17:26:42,001][WARN ][transport.netty ] [node1]
Message not fully read (response) for [19644702] handler
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$2@111480a2,
error [true], resetting

There is apparently no problem because the cluster is usable.
But since the 'Message not fully read' message appeared, i got 2 times
this message (on different index) :

[2013-02-27 17:37:33,647][WARN ][index.merge.scheduler ] [node1]
[index_name][0] failed to merge
org.apache.lucene.index.CorruptIndexException: docs out of order (6 <= 6 )
(out: org.apache.lucene.store.FSDirectory$FSIndexOutput@4d02bc28)
at
org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:84)
at
org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548)
at
org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4263)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

I have a 2 nodes ES cluster, each one is on 0.19.11 version.
Thank you for any piece of advice.
Germain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

What about your clients? Are you using Java clients of a different version?

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi clint,

I will check that but the name of the exception let me think that it's not
related to clients...

Le mercredi 6 mars 2013 20:01:33 UTC+1, Clinton Gormley a écrit :

What about your clients? Are you using Java clients of a different version?

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

When the problem was occurring, node2 was the master, i just tried to make
node1 becoming the new master and now, i have no more "Message not fully
read" message.
No clients were modified during this operation.

2013/3/7 Gmaurice gmaurice@linkfluence.net

Hi clint,

I will check that but the name of the exception let me think that it's not
related to clients...

Le mercredi 6 mars 2013 20:01:33 UTC+1, Clinton Gormley a écrit :

What about your clients? Are you using Java clients of a different
version?

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Q2sS58nT69Y/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-03-08 at 15:28 +0100, Germain Maurice wrote:

When the problem was occurring, node2 was the master, i just tried to
make node1 becoming the new master and now, i have no more "Message
not fully read" message.
No clients were modified during this operation.

What about different versions of Java on node1/node2?

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

node1 : java version "1.6.0_24"
node2 : java version "1.6.0_27"

A difference that could explain the behavior ?

2013/3/8 Clinton Gormley clint@traveljury.com

On Fri, 2013-03-08 at 15:28 +0100, Germain Maurice wrote:

When the problem was occurring, node2 was the master, i just tried to
make node1 becoming the new master and now, i have no more "Message
not fully read" message.
No clients were modified during this operation.

What about different versions of Java on node1/node2?

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Q2sS58nT69Y/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Fri, 2013-03-08 at 15:52 +0100, Germain Maurice wrote:

node1 : java version "1.6.0_24"

node2 : java version "1.6.0_27"

A difference that could explain the behavior ?

yes

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ok, i will fix that, thank you.

2013/3/8 Clinton Gormley clint@traveljury.com

On Fri, 2013-03-08 at 15:52 +0100, Germain Maurice wrote:

node1 : java version "1.6.0_24"

node2 : java version "1.6.0_27"

A difference that could explain the behavior ?

yes

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Q2sS58nT69Y/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Upgrading to the same version of java fixed the problem, I have no more
errors.

Thank you Clint !

2013/3/8 Germain Maurice gmaurice@linkfluence.net

Ok, i will fix that, thank you.

2013/3/8 Clinton Gormley clint@traveljury.com

On Fri, 2013-03-08 at 15:52 +0100, Germain Maurice wrote:

node1 : java version "1.6.0_24"

node2 : java version "1.6.0_27"

A difference that could explain the behavior ?

yes

clint

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Q2sS58nT69Y/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.