Installed marvel plugin to my elasticsearch, but the log gives:
[2015-11-29 15:59:52,514][ERROR][marvel.agent ] [crawler_service_001] background thread had an uncaught exception
ElasticsearchException[failed to flush exporter bulks]
at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:104)
at org.elasticsearch.marvel.agent.exporter.ExportBulk.close(ExportBulk.java:53)
at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:201)
at java.lang.Thread.run(Thread.java:745)
Suppressed: ElasticsearchException[failed to flush [default_local] exporter bulk]; nested: ElasticsearchException[failure in bulk execution, only the first 100 failures are printed:
[0]: index [.marvel-es-2015.11.29], type [indices_stats], id [AVFSQGxfYu-NU9LOfmGT], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[1]: index [.marvel-es-2015.11.29], type [cluster_stats], id [AVFSQGxfYu-NU9LOfmGU], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[2]: index [.marvel-es-2015.11.29], type [cluster_state], id [AVFSQGxfYu-NU9LOfmGV], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[3]: index [.marvel-es-2015.11.29], type [nodes], id [AVFSQGxfYu-NU9LOfmGW], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[4]: index [.marvel-es-data], type [node], id [Ax7rXEaMRjmHXT0wKn3BtA], message [UnavailableShardsException[[.marvel-es-data][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@29126d6d]]
[5]: index [.marvel-es-2015.11.29], type [shards], id [n5sRlr1tTiGh0B65lgPPOw:Ax7rXEaMRjmHXT0wKn3BtA:sfs-2015.11.12:2:p], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
[6]: index [.marvel-es-2015.11.29], type [shards], id [n5sRlr1tTiGh0B65lgPPOw:_na:sfs-2015.11.12:2:r], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]
//ignore some...
[99]: index [.marvel-es-2015.11.29], type [shards], id [n5sRlr1tTiGh0B65lgPPOw:Ax7rXEaMRjmHXT0wKn3BtA:.watch_history-2015.11.25:0:p], message [UnavailableShardsException[[.marvel-es-2015.11.29][0] Primary shard is not active or isn't assigned to a known node. Timeout: [1m], request: org.elasticsearch.action.bulk.BulkShardRequest@333878cf]]]
at org.elasticsearch.marvel.agent.exporter.local.LocalBulk.flush(LocalBulk.java:114)
at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:101)
... 3 more
And when I run curl -XGET http://localhost:9200/_cat/indices?v | grep marvel, I saw:
red open .marvel-es-2015.11.29 1 1
red open .marvel-es-data 1 1
Then curl -XGET http://localhost:9200/_cat/shards | grep marvel:
.marvel-es-2015.11.29 0 r UNASSIGNED
.marvel-es-data 0 p UNASSIGNED
.marvel-es-data 0 r UNASSIGNED
It seems elasticsearch didn't assign for node for marvel index, but I didn't know why.
I try to manually assign by:
curl -XPOST -d '{ "commands" : [ { "allocate" : { "index" : ".marvel-es-data", "shard" : 0, "node" :"crawler_service_001" } } ] }' http://172.16.11.17:9200/_cluster/reroute?pretty
results:
{
"error" : {
"root_cause" : [ {
"type" : "illegal_argument_exception",
"reason" : "[allocate] trying to allocate a primary shard [.marvel-es-data][0], which is disabled"
} ],
"type" : "illegal_argument_exception",
"reason" : "[allocate] trying to allocate a primary shard [.marvel-es-data][0], which is disabled"
},
"status" : 400
}
Environment:
- Elasticsearch - 2.1.0
- marvel-agent - 2.1.0