Hi all.
I have a dedicated Marvel cluster in my production environment. I noticed tonight that it was missing one of the nodes in my cluster. When I went to the missing data node (dedicated data only), I found this exception, seemingly out of the blue, repeated about 20 times:
[2015-12-01 22:09:11,685][ERROR][marvel.agent ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException: [J cannot be cast to [B
at org.apache.lucene.util.fst.BytesStore.ramBytesUsed(BytesStore.java:478)
at org.apache.lucene.util.fst.FST.ramBytesUsed(FST.java:435)
at org.apache.lucene.codecs.blocktree.FieldReader.ramBytesUsed(FieldReader.java:187)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.ramBytesUsed(BlockTreeTermsReader.java:309)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.ramBytesUsed(PerFieldPostingsFormat.java:238)
at org.apache.lucene.index.SegmentCoreReaders.ramBytesUsed(SegmentCoreReaders.java:226)
at org.apache.lucene.index.SegmentReader.ramBytesUsed(SegmentReader.java:614)
at org.elasticsearch.index.engine.Engine.segmentsStats(Engine.java:309)
at org.elasticsearch.index.shard.IndexShard.segmentStats(IndexShard.java:652)
at org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:175)
Then, every ~10 seconds, I'm now just getting this:
[2015-12-02 02:16:45,563][ERROR][marvel.agent ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:16:57,357][ERROR][marvel.agent ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:09,250][ERROR][marvel.agent ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:21,863][ERROR][marvel.agent ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:34,100][ERROR][marvel.agent ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:45,238][ERROR][marvel.agent ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:56,346][ERROR][marvel.agent ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:18:07,968][ERROR][marvel.agent ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
Not sure what happened, or what to do about it. Cycling the node is an obvious option, but I'd rather not take out a data node in production if I don't have to.
This cluster has been in prod for over a year, and is currently on ES 1.7.3, and Marvel 1.3.
Any ideas?
Thank you!
Chris