# Marvel.agent: failed to flush exporter bulks

**URL:** <https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817>\
**Category:** Elasticsearch\
**Created:** [November 29, 2015, 8:12am UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817 "2015-11-29T08:12:03Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Enniu\_51](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/enniu_51/32/5310_2.png) [@Enniu\_51](https://discuss.elastic.co/u/Enniu_51)\
**Post date:** [November 29, 2015, 8:12am UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/1 "2015-11-29T08:12:03Z")

</div>

Installed marvel plugin to my elasticsearch, but the log gives:

```
[2015-11-29 15:59:52,514][ERROR][marvel.agent] [crawler_service_001] background thread had an uncaught exception
ElasticsearchException[failed to flush exporter bulks]
	at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:104)
	at org.elasticsearch.marvel.agent.exporter.ExportBulk.close(ExportBulk.java:53)
	at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:201)
	at java.lang.Thread.run(Thread.java:745)
	Suppressed: ElasticsearchException[failed to flush [default_local] exporter bulk]; nested: ElasticsearchException[failure in bulk execution, only the first 100 failures are printed:
[0]: index [.marvel-es-2015.11.29], type [indices_stats], id [AVFSQGxfYu-NU9LOfmGT], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[1]: index [.marvel-es-2015.11.29], type [cluster_stats], id [AVFSQGxfYu-NU9LOfmGU], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[2]: index [.marvel-es-2015.11.29], type [cluster_state], id [AVFSQGxfYu-NU9LOfmGV], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[3]: index [.marvel-es-2015.11.29], type [nodes], id [AVFSQGxfYu-NU9LOfmGW], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[4]: index [.marvel-es-data], type [node], id [Ax7rXEaMRjmHXT0wKn3BtA], message [UnavailableShardsException[[.marvel-es-data][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@29126d6d]]
[5]: index [.marvel-es-2015.11.29], type [shards], id [n5sRlr1tTiGh0B65lgPPOw:Ax7rXEaMRjmHXT0wKn3BtA:sfs-2015.11.12:2:p], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[6]: index [.marvel-es-2015.11.29], type [shards], id [n5sRlr1tTiGh0B65lgPPOw:_na:sfs-2015.11.12:2:r], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
//ignore some...
[99]: index [.marvel-es-2015.11.29], type [shards], id [n5sRlr1tTiGh0B65lgPPOw:Ax7rXEaMRjmHXT0wKn3BtA:.watch_history-2015.11.25:0:p], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]]
		at org.elasticsearch.marvel.agent.exporter.local.LocalBulk.flush(LocalBulk.java:114)
		at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:101)
		... 3 more

```

And when I run `curl -XGET http://localhost:9200/_cat/indices?v | grep marvel`, I saw:

```
red open .marvel-es-2015.11.29 1 1                                                   
red open .marvel-es-data 1 1                                                   

```

Then `curl -XGET http://localhost:9200/_cat/shards | grep marvel`:

```
.marvel-es-2015.11.29 0 r UNASSIGNED                                              
.marvel-es-data 0 p UNASSIGNED                                              
.marvel-es-data 0 r UNASSIGNED  

```

It seems elasticsearch didn't assign for node for marvel index, but I didn't know why.

I try to manually assign by:

```
curl -XPOST -d '{ "commands" : [{ "allocate" : { "index" : ".marvel-es-data", "shard" : 0, "node" :"crawler_service_001" } }] }' http://172.16.11.17:9200/_cluster/reroute?pretty

```

results:

```
{
   "error" : {
     "root_cause" : [ {
        "type" : "illegal_argument_exception",
        "reason" : "[allocate] trying to allocate a primary shard [.marvel-es-data][0], which is disabled"
    } ],
    "type" : "illegal_argument_exception",
    "reason" : "[allocate] trying to allocate a primary shard [.marvel-es-data][0], which is disabled"
  },
  "status" : 400
}

```

Environment:

- Elasticsearch - 2.1.0
- marvel-agent - 2.1.0

---

<div class="post-metadata">

**Author:** ![Enniu\_51](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/enniu_51/32/5310_2.png) [@Enniu\_51](https://discuss.elastic.co/u/Enniu_51)\
**Post date:** [November 30, 2015, 7:28am UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/2 "2015-11-30T07:28:10Z")

</div>

Okay, I moved all my indices to another place, and it worked...

And I will reimport the indices by logstash...

But I'm still curious about what happened to my elasticsearch and lead it to into such a state...

---

<div class="post-metadata">

**Author:** ![elain](https://avatars.discourse-cdn.com/v4/letter/e/a5b964/32.png) [@elain](https://discuss.elastic.co/u/elain)\
**Post date:** [December 25, 2015, 8:57am UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/3 "2015-12-25T08:57:04Z")

</div>

I've had the same problem.

elasticssearch 2.1.1  
kibana 4.3.1

---

<div class="post-metadata">

**Author:** ![voipoclay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voipoclay/32/4372_2.png) [@voipoclay](https://discuss.elastic.co/u/voipoclay)\
**Post date:** [January 7, 2016, 6:05pm UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/4 "2016-01-07T18:05:01Z")

</div>

I have same issue here.

```
curl -XGET http://eth1:9200/_cat/shards -s | grep marvel
.marvel-es-2016.01.07 0 r STARTED 2077820 977.6mb 192.241.221.72 NODE_6 
.marvel-es-2016.01.07 0 p STARTED 2077820 965.8mb 192.241.208.110 NODE_2 
.marvel-es-data 0 p STARTED 15 8.9kb 192.241.221.72 NODE_6 
.marvel-es-data 0 r STARTED 15 11.3kb 192.241.208.110 NODE_2 

```

Mine seems to kinda load intermittently. I think its the shard activity on bottom having issues.

I have 4200 shards and 420 Indices. Any thoughts?

---

<div class="post-metadata">

**Author:** ![flyingrabbit](https://avatars.discourse-cdn.com/v4/letter/f/848f3c/32.png) [@flyingrabbit](https://discuss.elastic.co/u/flyingrabbit)\
**Post date:** [March 16, 2016, 3:05pm UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/6 "2016-03-16T15:05:47Z")

</div>

[2016-03-16 09:52:57,944][ERROR][marvel.agent] [cs-mv1-agtfs] background thread had an uncaught exception  
ElasticsearchException[failed to flush exporter bulks]  
at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:104)  
at org.elasticsearch.marvel.agent.exporter.ExportBulk.close(ExportBulk.java:53)  
at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:201)  
at java.lang.Thread.run(Thread.java:745)  
Suppressed: ElasticsearchException[failed to flush [default\_local] exporter bulk]; nested: ElasticsearchException[failure in bulk execution:  
[0]: index [.marvel-es-2016.03.16], type [indices\_stats], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[1]: index [.marvel-es-2016.03.16], type [index\_stats], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[2]: index [.marvel-es-2016.03.16], type [index\_stats], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[3]: index [.marvel-es-2016.03.16], type [index\_stats], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[4]: index [.marvel-es-2016.03.16], type [index\_stats], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[5]: index [.marvel-es-2016.03.16], type [index\_stats], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[6]: index [.marvel-es-2016.03.16], type [cluster\_state], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[7]: index [.marvel-es-2016.03.16], type [nodes], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[9]: index [.marvel-es-2016.03.16], type [nodes], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[11]: index [.marvel-es-2016.03.16], type [nodes], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[13]: index [.marvel-es-2016.03.16], type [nodes], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[15]: index [.marvel-es-2016.03.16], type [index\_recovery], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]  
[16]: index [.marvel-es-2016.03.16], type [cluster\_stats], id [null], message [[.marvel-es-2016.03.16] IndexNotFoundException[no such index]]

---

<div class="post-metadata">

**Author:** ![flyingrabbit](https://avatars.discourse-cdn.com/v4/letter/f/848f3c/32.png) [@flyingrabbit](https://discuss.elastic.co/u/flyingrabbit)\
**Post date:** [March 16, 2016, 3:10pm UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/7 "2016-03-16T15:10:16Z")

</div>

The weird thing is I tried exporting locally and it worked. It seems like a communication issue between the production cluster and the monitoring cluster. This is marvel 2.1.2.  
I have tried pointing to the monitoring server cluster with FQDN, IP, Hostname.

I can pull up head from any server in the production node pointing at the monitoring node and vice versa. So I don't think it is a firewall issue.

Anyone seen anything like this?

---

<div class="post-metadata">

**Author:** ![flyingrabbit](https://avatars.discourse-cdn.com/v4/letter/f/848f3c/32.png) [@flyingrabbit](https://discuss.elastic.co/u/flyingrabbit)\
**Post date:** [March 16, 2016, 3:15pm UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/8 "2016-03-16T15:15:52Z")

</div>

I'm just wondering why this entry appears:  
Suppressed: ElasticsearchException[failed to flush [default\_local] exporter bulk]; nested: ElasticsearchException[failure in bulk execution:  
It seems like its attempting to write locally. My setting is as follows:  
marvel.agent.exporters:  
id1:  
type: http  
host: ["[http://NAMEOFTHEMONITORNODE:9200](http://NAMEOFTHEMONITORNODE:9200)"]

---

<div class="post-metadata">

**Author:** ![flyingrabbit](https://avatars.discourse-cdn.com/v4/letter/f/848f3c/32.png) [@flyingrabbit](https://discuss.elastic.co/u/flyingrabbit)\
**Post date:** [March 16, 2016, 3:22pm UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/9 "2016-03-16T15:22:03Z")

</div>

I also tried making the marvel indexes manually on the monitoring server and they don't get populated. If I make the marvel indexes on the production cluster they do get populated. It's seems to be ignoring the setting for the marvel agent completely.

---

<div class="post-metadata">

**Author:** ![pickypg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pickypg/32/62409_2.png) [@pickypg](https://discuss.elastic.co/u/pickypg)\
**Post date:** [May 7, 2016, 12:23am UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/10 "2016-05-07T00:23:39Z")

</div>

I saw this occur recently for a user that had messed up their cluster. If this occurs, please verify that your Marvel indices are not red:

```auto
$ curl -XGET host:9200/_cat/indices/.marvel*?v

```

In the user's case, their `.marvel-data-1` index was `red` because they had lost all of their shards, which was causing this exception to be triggered over and over again.

---

<div class="post-metadata">

**Author:** ![memelet](https://avatars.discourse-cdn.com/v4/letter/m/6bbea6/32.png) [@memelet](https://discuss.elastic.co/u/memelet)\
**Post date:** [July 15, 2016, 1:49am UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/11 "2016-07-15T01:49:24Z")

</div>

We just got these across our cluster. 7 out of 10 data nodes dropped out of the cluster shortly after these exceptions.

(We are running marvel on the same cluster. ES 2.3.3)

---

<div class="post-metadata">

**Author:** ![dfaropennetwork](https://avatars.discourse-cdn.com/v4/letter/d/9fc29f/32.png) [@dfaropennetwork](https://discuss.elastic.co/u/dfaropennetwork)\
**Post date:** [August 25, 2016, 3:23am UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/12 "2016-08-25T03:23:05Z")

</div>

Hi, i had the same problem, my solution:

> 

~$ curl -u admin -XGET [http://hsotname:9200/\_cat/indices?v](http://hsotname:9200/_cat/indices?v) | grep marvel (or red or yellow)  
red open .marvel-es-data-1 1 1  
yellow open .marvel-es-1-2016.08.25 1 1 10837 1380 3.8mb 3.8mb  
yellow open .marvel-es-1-2016.08.04 1 1 24934 1592 5.9mb 5.9mb  
red open .marvel-es-1-2016.08.03 1 1

This is not necessary for my, i deleted

> curl -u admin -XDELETE '[http://hostname:9200/.marvel-es-1-2016.08.03/](http://hostname:9200/.marvel-es-1-2016.08.03/)'  
> {"acknowledged":true}

And I remove the replicas:

> $ curl -u admin -XPUT 'hostname:9200/\_settings' -d '  
> {  
> "index" : {  
> "number\_of\_replicas" : 0  
> }  
> }'

Now my cluster status is green.

---

<div class="post-metadata">

**Author:** ![seth.yes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seth.yes/32/11788_2.png) [@seth.yes](https://discuss.elastic.co/u/seth.yes)\
**Post date:** [October 26, 2016, 3:06pm UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/13 "2016-10-26T15:06:01Z")

</div>

I've had the same issue, fix for me was as mentioned prior -- move your data to another location.. must be a perms issue?

---

<div class="post-metadata">

**Author:** ![Jules\_Huang](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jules\_Huang](https://discuss.elastic.co/u/Jules_Huang)\
**Post date:** [March 6, 2017, 8:36pm UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/14 "2017-03-06T20:36:36Z")

</div>

I have the same issue. I have three indices were marked as red .marvel-es-1-2017.03.04, .marvel-es-1-2017.03.05, and .marvel-es-1-2017.03.06. I deleted the first two with no issues. But when I deleted the first one, which is for today, one was auto created and still with red. Any idea how I can fix this?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:02pm UTC](https://discuss.elastic.co/t/marvel-agent-failed-to-flush-exporter-bulks/35817/15 "2017-07-05T22:02:32Z")

</div>