Marvel index rate maybe high



we put data into es by logstash from kafka, kafka consume rate is about 300message/s and logstash metrics plugin calculate is about same with kafka ,but es index rate is 675.2/s

our index is 2 shard 1 replicas
es 2.1 kibana 4.3+
if can tell me how to get the index rate by es restful request,i try to use "index/_stats" and find "index_current",but the value is dramatically different

(Tanguy) #2


Marvel takes replicas into account when calculating indexing rate. In your case, every write goes to 1 primary shard and in 1 replica so the 300 index/sec are in fact in total 600 index / sec in your cluster.

We are thinking of changing the way the indexing rate is calculated. The Index Stats (see can be used to know the number of index operations that have been executed on primary shards only (see primaries.indexing.index_total).


thinks for your reply ,i have get the value by "curl localhost:9200/index/_stats?pretty",now marvel index rate is 233.67/s,primaries.indexing.index_total is 8865237, i don't know relationship between "index rate" and "primaries.indexing.index_total",can you tell me ,thank you very much:grin:

(Michal) #4

@whybangbang: could you tell me what application is used to show messages in/sec?
This is additional kibana dashboard or somthing else?
How do you do kafka benchmark in logstash? Using metric codec/filter?


"kafka manager" show message/s and "metric filter" show logstash message out/sec

(system) #6