# Marvel.agent.exprorer create failure kills cluster
**URL:** https://discuss.elastic.co/t/marvel-agent-exprorer-create-failure-kills-cluster/47239
**Category:** Elasticsearch
**Tags:** elastic-stack-monitoring
**Created:** [April 13, 2016, 11:24am UTC](https://discuss.elastic.co/t/marvel-agent-exprorer-create-failure-kills-cluster/47239 "2016-04-13T11:24:56Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Volodymyr\_Bilyachat](https://discuss.elastic.co/u/Volodymyr_Bilyachat)
#### Post date: [April 13, 2016, 11:24am UTC](https://discuss.elastic.co/t/marvel-agent-exprorer-create-failure-kills-cluster/47239/1 "2016-04-13T11:24:56Z")
Hi all,
I have huge problem that when marvel cant create index it actually kills cluster.
I am using version 1.4.5 and last month we have strange issue when almost everyweek cluster dies. Each time there is in logs
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
[2016-04-13 12:58:48,230][ERROR][marvel.agent.exporter] [Node1] create failure (index:[.marvel-2016.04.13] type: [index\_stats]): RemoteTransportException[[Node4][inet[/IP:9300]][indices:data/write/bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsea$
---
### Author:  [@skearns](https://discuss.elastic.co/u/skearns)
#### Post date: [April 13, 2016, 3:26pm UTC](https://discuss.elastic.co/t/marvel-agent-exprorer-create-failure-kills-cluster/47239/2 "2016-04-13T15:26:53Z")
Hi Volodymyr,
It looks like your cluster is overloaded. Those exceptions are saying that your bulk indexing queues are full and the cluster cannot process the indexing requests from Marvel. Given that Marvel has a consistent indexing rate (every 10 seconds by default), I would imagine that there are other things going on here that are causing the issues.
Do you perform bulk indexing actions in other processes?
Can you confirm which version of ES, and which version of Marvel you are running?
Thanks,
Steve
---
### Author:  [@pickypg](https://discuss.elastic.co/u/pickypg)
#### Post date: [April 13, 2016, 3:27pm UTC](https://discuss.elastic.co/t/marvel-agent-exprorer-create-failure-kills-cluster/47239/3 "2016-04-13T15:27:03Z")
Hi,
So you're running ES 1.4.5, but which version of Marvel?
The errors that you're seeing relate to bulk ingestion being rejected -- it means that the bulk queue in your nodes is backed up when Marvel tries to send more stats (`index_stats` in this case). Specifically, it looks like `Node4` is the one backed up.
This means that that node is too busy to handle any additional bulk ingestion. Take a look at that node to see why it's backing up.
Hope that helps,
Chris
---
### Author:  [@Volodymyr\_Bilyachat](https://discuss.elastic.co/u/Volodymyr_Bilyachat)
#### Post date: [April 14, 2016, 7:16am UTC](https://discuss.elastic.co/t/marvel-agent-exprorer-create-failure-kills-cluster/47239/4 "2016-04-14T07:16:05Z")
I dont understand versions in Marvel so i suppose its 1.3 because i cant find any version.
As i see this is issue with ES, so next question is is this there any way to kind of skip bulk if there is too much work? because for me problem is that cluster dies and to wake it up i need to restart it.
---
### Author:  [@skearns](https://discuss.elastic.co/u/skearns)
#### Post date: [April 14, 2016, 12:04pm UTC](https://discuss.elastic.co/t/marvel-agent-exprorer-create-failure-kills-cluster/47239/5 "2016-04-14T12:04:49Z")
Hi Volodymyr,
Given that this looks like a production cluster, I recommend that you setup a separate Monitoring cluster, and have Marvel send it's data to the monitoring cluster, so your monitoring data is on a different cluster than the cluster being monitored. This should reduce the load on your production cluster, and will make it easier to troubleshoot any issues with your production cluster.
To do this, you need to configure the Marvel Exporter to point at a different cluster. If you don't configure it (the default), it will store monitoring data in the local cluster.
[https://www.elastic.co/guide/en/marvel/marvel-1.3/stats-export.html#stats-export](https://www.elastic.co/guide/en/marvel/marvel-1.3/stats-export.html#stats-export)
Thanks,
Steve
---
### Author:  [@Volodymyr\_Bilyachat](https://discuss.elastic.co/u/Volodymyr_Bilyachat)
#### Post date: [April 15, 2016, 11:13am UTC](https://discuss.elastic.co/t/marvel-agent-exprorer-create-failure-kills-cluster/47239/6 "2016-04-15T11:13:32Z")
Thank you i will start from doing that, also i have some logging to production cluster so i will switch that to preproduction
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:44pm UTC](https://discuss.elastic.co/t/marvel-agent-exprorer-create-failure-kills-cluster/47239/7 "2017-07-06T13:44:20Z")