Marvel index rate maybe high

[whybangbang]

we put data into es by logstash from kafka, kafka consume rate is about 300message/s and logstash metrics plugin calculate is about same with kafka ,but es index rate is 675.2/s

our index is 2 shard 1 replicas
es 2.1 kibana 4.3+
if can tell me how to get the index rate by es restful request,i try to use "index/_stats" and find "index_current",but the value is dramatically different

[tanguy]

Hi,

Marvel takes replicas into account when calculating indexing rate. In your case, every write goes to 1 primary shard and in 1 replica so the 300 index/sec are in fact in total 600 index / sec in your cluster.

We are thinking of changing the way the indexing rate is calculated. The Index Stats (see https://www.elastic.co/guide/en/elasticsearch/reference/2.1/indices-stats.html) can be used to know the number of index operations that have been executed on primary shards only (see primaries.indexing.index_total).

[whybangbang]

thinks for your reply ,i have get the value by "curl localhost:9200/index/_stats?pretty",now marvel index rate is 233.67/s,primaries.indexing.index_total is 8865237, i don't know relationship between "index rate" and "primaries.indexing.index_total",can you tell me ,thank you very much:grin:

[michalterbert (Michal)]

@whybangbang: could you tell me what application is used to show messages in/sec?
This is additional kibana dashboard or somthing else?
How do you do kafka benchmark in logstash? Using metric codec/filter?

[whybangbang]

"kafka manager" show message/s and "metric filter" show logstash message out/sec