I notice in marvel the highest insert activity belongs to the marvel index and shows around 680/s to 960/s.
I was curious why this is so high?
Hi voipoclay,
That seems to be high indeed. Which exact versions are you using? How many nodes / indices / shards do you cluster have?
ES 2.1.1
Kibana 4.3.0
Marvel (not sure how to tell)
Nodes: 11
Indices: 438
Memory: 11GB / 57GB
Total Shards: 4348
Unassigned Shards: 0
Documents: 45,107,944
Data: 35GB
Marvel is indexing a set of documents for every nodes, indices and shards of your cluster. These documents are used to build the various charts.
In your case, you have many nodes / indices /shards compared to the volume of data (35 GB). That's around 100K / 80MB per index and this is not a lot.
Every 10 sec you have 11 (nodes) + 438 (indices) + 4348 (shards) = ~4800 documents indexed in 2 shards (marvel indices), so a total of 9600 docs every 10 secs which explains 960 docs /sec.
Hope this helps
Ok thanks for the detailed reply and I can see how this makes sense now.
I wish that I could monitor the cluster without such a load. It seem like 1000/s when the rest of my cluster is fairly idle seems like a lot of load to monitor and kind of gives me "false" readouts in a sense because 90% of the load is just from marvel and not actual production activity.
I guess I can live with it since it does have useful information. Would be kind of nice if there was a way to store the metrics for marvel locally to the kibana machine rather then on the target cluster.
Regardless thank you for explaining this to me!
The marvel agent plugin can be configured to export data to another monitoring cluster (using a HTTP Exporter) so that marvel data are not stored locally in the same cluster as your business data. This is the recommended configuration for production clusters.
In your case, I'd also investigate how to reduce the number of indices & shards.
enter link description here
if you can help me solve the problem,thank you, as this case ,i also find index rate is high