I started getting this around midnight:
[2015-11-25 00:00:13,872][WARN ][http.netty ] [HEALTH_NODE_1] Caught exception while handling client http traffic, closing connection [id: 0xc3c2bfc2, /10.1.55.2:48602 => /10.1.55.21:9200]
[2015-11-25 09:23:20,873][INFO ][rest.suppressed ] /_bulk Params: {}
ElasticsearchParseException[Failed to derive xcontent]
I stopped all my ES nodes in my monitored cluster (to do some other maintenance) and when I restarted them I had marvel showing me data, but today when I cam back I am getting the above error.
EDIT***
Further research shows the following on my master nodes in my monitored cluster:
[2015-11-25 10:19:11,695][ERROR][marvel.agent.exporter.http] failed sending data to [http://10.1.55.22:9200/_bulk]: IOException[Error writing to server] [2015-11-25 10:19:11,695][ERROR][marvel.agent ] [MASTER_NODE_1] background thread had an uncaught exception java.lang.OutOfMemoryError: Java heap space
Currently my master nodes have 32G of ram with 24 set in /etc/sysconfig/elasticsearch:
ES_HEAP_SIZE=24g
I'm working on getting all nodes updated to 64 gig with 30 set to ES_HEAP_SIZE