# \[marvel.agent \] background thread had an uncaught exception ElasticsearchException\[failed to flush exporter bulks on today's index **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-monitoring\ **Created:** [March 6, 2017, 7:40pm UTC](https://discuss.elastic.co/t/marvel-agent-background-thread-had-an-uncaught-exception-elasticsearchexception-failed-to-flush-exporter-bulks-on-todays-index/77571 "2017-03-06T19:40:20Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Jules\_Huang](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jules\_Huang](https://discuss.elastic.co/u/Jules_Huang)\ **Post date:** [March 6, 2017, 7:40pm UTC](https://discuss.elastic.co/t/marvel-agent-background-thread-had-an-uncaught-exception-elasticsearchexception-failed-to-flush-exporter-bulks-on-todays-index/77571/1 "2017-03-06T19:40:20Z")
I started with elasticsearch 1.6.0. To upgraded it to 5.2 based on the recommendation that I upgraded it to 2.4.4 first. As of now I am still at 2.4 with Kibana 4.6.4. I have one node PerfNode\_1 with the data migrated from 1.6.0. The node has been complaining .marvel-es-1-2017-03.06 is red and the rest indices are yellow. After some googling I decided to delete it and let it recreated. But once it's recreated it's still red. Because of that Marvel is not receiving data either. This is my log: 017.03.06][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest to [.marvel-es-1-2017.03.06] containing [166] requests]]]]; at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:106) ... 3 more Caused by: ElasticsearchException[failure in bulk execution, only the first 100 failures are printed: [0]: index [.marvel-es-1-2017.03.06], type [index\_stats], id [AVqlGSdIF4aaLd2WG6c7], message [UnavailableShardsException[[.marvel-es-1-2017.03.06][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest to [.marvel-es-1-2017.03.06] containing [166] requests]]] [1]: index [.marvel-es-1-2017.03.06], type [index\_stats], id [AVqlGSdIF4aaLd2WG6c8], message [UnavailableShardsException[[.marvel-es-1-2017.03.06][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest to [.marvel-es-1-2017.03.06] containing [166] requests]]] This is my cluster indices: [tw@ultva02 kibana]$ curl -XGET 10.170.98.33:9200/\_cat/indices?pretty yellow open .marvel-2017.02.08 1 1 11865 0 16.7mb 16.7mb yellow open .marvel-2017.02.09 1 1 143172 0 191.7mb 191.7mb yellow open .marvel-2017.02.28 1 1 293865 0 339.6mb 339.6mb yellow open .marvel-2017.02.06 1 1 119778 0 170.6mb 170.6mb yellow open .marvel-2017.02.07 1 1 127153 0 174.3mb 174.3mb yellow open .marvel-2017.02.04 1 1 102794 0 153.2mb 153.2mb yellow open .marvel-2017.02.26 1 1 280068 0 326.7mb 326.7mb yellow open .marvel-2017.02.27 1 1 287036 0 334mb 334mb yellow open .marvel-2017.02.05 1 1 111321 0 162.7mb 162.7mb yellow open .marvel-2017.02.24 1 1 258614 0 304.5mb 304.5mb yellow open .marvel-2017.02.02 1 1 1579356 0 737.2mb 737.2mb yellow open .marvel-2017.02.25 1 1 272785 0 319.3mb 319.3mb yellow open .marvel-2017.02.03 1 1 92276 0 138.9mb 138.9mb yellow open .marvel-2017.02.22 1 1 238327 0 242.4mb 242.4mb yellow open .marvel-2017.02.01 1 1 887400 0 387.4mb 387.4mb yellow open .marvel-2017.02.23 1 1 250149 0 294.5mb 294.5mb yellow open .marvel-2017.02.20 1 1 220623 0 208.4mb 208.4mb yellow open .marvel-2017.02.21 1 1 228373 0 211.9mb 211.9mb yellow open .marvel-es-1-2017.03.02 1 1 69228 580 22.8mb 22.8mb yellow open .marvel-es-1-2017.03.03 1 1 372978 944 139.5mb 139.5mb yellow open .kibana 1 1 2 0 8.9kb 8.9kb yellow open .marvel-es-data-1 1 1 22 1 2.4mb 2.4mb yellow open .marvel-kibana 5 1 2 1 20.4kb 20.4kb yellow open staging 5 1 395164 0 78.5mb 78.5mb yellow open blotterdata 5 1 3963291 0 981.7mb 981.7mb yellow open .marvel-2017.02.19 1 1 212739 0 202.1mb 202.1mb yellow open .marvel-2017.02.17 1 1 197985 0 190.1mb 190.1mb yellow open .marvel-2017.02.18 1 1 205533 0 196.8mb 196.8mb yellow open .marvel-2017.02.15 1 1 182193 0 177.6mb 177.6mb yellow open index 5 1 4 0 12.9kb 12.9kb yellow open .marvel-2017.02.16 1 1 189990 0 183.3mb 183.3mb yellow open .marvel-2017.02.13 1 1 166753 0 164.8mb 164.8mb yellow open .marvel-2017.02.14 1 1 174459 0 171.7mb 171.7mb yellow open .marvel-2017.02.11 1 1 150836 0 152.4mb 152.4mb yellow open .marvel-2017.03.01 1 1 303346 0 317.3mb 317.3mb yellow open .marvel-2017.03.02 1 1 214931 0 185.7mb 185.7mb yellow open .marvel-2017.02.12 1 1 158835 0 159mb 159mb red open .marvel-es-1-2017.03.06 1 1 yellow open .marvel-2017.02.10 1 1 148360 0 182.6mb 182.6mb yellow open report 5 1 133623382 0 21.7gb 21.7gb My elasticsearch.yml. The other node has exactly the same except [node.name](http://node.name) and node.host. the rest are defaults. [cluster.name](http://cluster.name): elasticsearch\_tw [node.name](http://node.name): "PerfNode\_1" node.rack: PerfNode\_1 path.data: /home/elasticsearch-1.6.0/data indices.recovery.max\_bytes\_per\_sec: 500mb bootstrap.mlockall: true network.host: 10.170.98.33 cluster.routing.allocation.awareness.attributes: rack http.cors.enabled: true http.cors.allow-origin: /.\*/ http.cors.allow-credentials: true path.repo: /home/elasticsearch-1.6.0/data/backup ---
**Author:** ![Jules\_Huang](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jules\_Huang](https://discuss.elastic.co/u/Jules_Huang)\ **Post date:** [March 6, 2017, 8:56pm UTC](https://discuss.elastic.co/t/marvel-agent-background-thread-had-an-uncaught-exception-elasticsearchexception-failed-to-flush-exporter-bulks-on-todays-index/77571/2 "2017-03-06T20:56:26Z")
I followed this [https://github.com/elastic/elasticsearch/issues/16708](https://github.com/elastic/elasticsearch/issues/16708) and run allocate on the last index curl -XPOST [http://localhost:9200/\_cluster/reroute?pretty](http://localhost:9200/_cluster/reroute?pretty) -d '{ "commands" : [{ "allocate" : { "index" : ".marvel-es-data", "shard" : 0, "node" :"i-e098e03a", "allow\_primary": "true" } }] }' Now my indices are all yellow and my Marvel works again. ---
**Author:** ![pickypg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pickypg/32/62409_2.png) [@pickypg](https://discuss.elastic.co/u/pickypg)\ **Post date:** [March 7, 2017, 12:25am UTC](https://discuss.elastic.co/t/marvel-agent-background-thread-had-an-uncaught-exception-elasticsearchexception-failed-to-flush-exporter-bulks-on-todays-index/77571/3 "2017-03-07T00:25:40Z")
Glad you were able to figure it out. FYI, you can go ahead and delete all of the old Marvel indices: ```auto curl -XDELETE 'http://localhost:9200/.marvel-2017*' ``` They won't be useful to you anymore. Good luck, Chris ---
**Author:** ![Jules\_Huang](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jules\_Huang](https://discuss.elastic.co/u/Jules_Huang)\ **Post date:** [March 7, 2017, 3:03pm UTC](https://discuss.elastic.co/t/marvel-agent-background-thread-had-an-uncaught-exception-elasticsearchexception-failed-to-flush-exporter-bulks-on-todays-index/77571/4 "2017-03-07T15:03:09Z")
Is the new indices named as .marvel-es-1-yyyy-mm.dd? If that's the case I should delete all old .marvel-2017\* indices instead of migrating them? ---
**Author:** ![pickypg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pickypg/32/62409_2.png) [@pickypg](https://discuss.elastic.co/u/pickypg)\ **Post date:** [March 7, 2017, 5:15pm UTC](https://discuss.elastic.co/t/marvel-agent-background-thread-had-an-uncaught-exception-elasticsearchexception-failed-to-flush-exporter-bulks-on-todays-index/77571/5 "2017-03-07T17:15:18Z")
@Jules_Huang Exactly! There is no benefit to keeping around your Marvel 1.x indices. Also, given the intent to try to get to 5.x, indices from 1.x are incompatible with 5.x, so deleting them solves that issue as well. Hope that helps, Chris ---
**Author:** ![Jules\_Huang](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@Jules\_Huang](https://discuss.elastic.co/u/Jules_Huang)\ **Post date:** [March 7, 2017, 6:43pm UTC](https://discuss.elastic.co/t/marvel-agent-background-thread-had-an-uncaught-exception-elasticsearchexception-failed-to-flush-exporter-bulks-on-todays-index/77571/6 "2017-03-07T18:43:01Z")
Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 4, 2017, 6:43pm UTC](https://discuss.elastic.co/t/marvel-agent-background-thread-had-an-uncaught-exception-elasticsearchexception-failed-to-flush-exporter-bulks-on-todays-index/77571/7 "2017-04-04T18:43:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.