Strange Marvel exception

monitoring

(Chris Neal) #1

Hi all.

I have a dedicated Marvel cluster in my production environment. I noticed tonight that it was missing one of the nodes in my cluster. When I went to the missing data node (dedicated data only), I found this exception, seemingly out of the blue, repeated about 20 times:

[2015-12-01 22:09:11,685][ERROR][marvel.agent             ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException: [J cannot be cast to [B
at org.apache.lucene.util.fst.BytesStore.ramBytesUsed(BytesStore.java:478)
at org.apache.lucene.util.fst.FST.ramBytesUsed(FST.java:435)
at org.apache.lucene.codecs.blocktree.FieldReader.ramBytesUsed(FieldReader.java:187)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.ramBytesUsed(BlockTreeTermsReader.java:309)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.ramBytesUsed(PerFieldPostingsFormat.java:238)
at org.apache.lucene.index.SegmentCoreReaders.ramBytesUsed(SegmentCoreReaders.java:226)
at org.apache.lucene.index.SegmentReader.ramBytesUsed(SegmentReader.java:614)
at org.elasticsearch.index.engine.Engine.segmentsStats(Engine.java:309)
at org.elasticsearch.index.shard.IndexShard.segmentStats(IndexShard.java:652)
at org.elasticsearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:175)

Then, every ~10 seconds, I'm now just getting this:

[2015-12-02 02:16:45,563][ERROR][marvel.agent             ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:16:57,357][ERROR][marvel.agent             ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:09,250][ERROR][marvel.agent             ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:21,863][ERROR][marvel.agent             ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:34,100][ERROR][marvel.agent             ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:45,238][ERROR][marvel.agent             ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:17:56,346][ERROR][marvel.agent             ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException
[2015-12-02 02:18:07,968][ERROR][marvel.agent             ] [elasticsearch-bdprodes07] Background thread had an uncaught exception:
java.lang.ClassCastException

Not sure what happened, or what to do about it. Cycling the node is an obvious option, but I'd rather not take out a data node in production if I don't have to.

This cluster has been in prod for over a year, and is currently on ES 1.7.3, and Marvel 1.3.

Any ideas?

Thank you!
Chris


(Tanguy) #2

Hi,

Which JVM (version/vendor) are you running?


(Chris Neal) #3

This one:

chris.neal@bdprodes07:[59]:~> java -version
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)

Thanks!
Chris


(system) #4