Indices total indexing time metric given by marvel


(sindhu) #1

Hi,

I'm new to elastic search and marvel . I'm currently investigating marvel
and I want to know if the "Indices total indexing time" shown by marvel is
"sum of the times taken by each shard" or "time between the arrival of
request and completion of indexing".

For example if there are 5 shards and indexing request arrives at 's' and
indexing completes at 'e'. And shards take time t1,t2,t3,t4 and t5
respectively. So how is Indices total indexing time calculated now. Is ut
e-s or t1+t2+t3+t4+t5.

To add to this question, just want to know if marvel uses stats api.

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/97700530-a3e7-4d72-9a37-31ec85161c10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Boaz Leskes) #2

Hi,

By the title, I assume you refer to the chart show in the Node Stats
dashboard. This one is tricky. It shows you the total spent on indexing per
second. This is best explained by an example - say you have 2 shards on a
node. Say that each indexes 100% of the time (continuously indexing) and
they do so in parallel. In this case we have 2 seconds of indexing (total
of two shards) for each real-world second. In other words - see this as a
load factor of indexing on the node.

Technically we indeed use the node stats api which gives you the total
spent on indexing of all shards that ever were on this node (reset on
restart).In your example it will be t1+t2+t3+t4 (if all shards were on that
node). The chart you see then plots the derivative (change per second) of
this value.

Hope this helps,
Boaz

On Wednesday, April 16, 2014 11:25:38 AM UTC+2, sindhu wrote:

Hi,

I'm new to elastic search and marvel . I'm currently investigating marvel
and I want to know if the "Indices total indexing time" shown by marvel is
"sum of the times taken by each shard" or "time between the arrival of
request and completion of indexing".

For example if there are 5 shards and indexing request arrives at 's' and
indexing completes at 'e'. And shards take time t1,t2,t3,t4 and t5
respectively. So how is Indices total indexing time calculated now. Is ut
e-s or t1+t2+t3+t4+t5.

To add to this question, just want to know if marvel uses stats api.

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8c04d37e-1b69-44de-af7c-ab94ac474cfc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(sindhu) #3

Hi,

Thanks for the repy. Helped a lot. I also have a query about index rate
total. Is it

  1. no of documents indexed per sec
  2. no of indexing req coming per sec
  3. no of indexing requests completed per sec

Can you also provide any links to documentation about what each metric
exactly represents if possible.

Thanks,
sindhu.

On Thu, Apr 17, 2014 at 1:30 AM, Boaz Leskes b.leskes@gmail.com wrote:

Hi,

By the title, I assume you refer to the chart show in the Node Stats
dashboard. This one is tricky. It shows you the total spent on indexing per
second. This is best explained by an example - say you have 2 shards on a
node. Say that each indexes 100% of the time (continuously indexing) and
they do so in parallel. In this case we have 2 seconds of indexing (total
of two shards) for each real-world second. In other words - see this as a
load factor of indexing on the node.

Technically we indeed use the node stats api which gives you the total
spent on indexing of all shards that ever were on this node (reset on
restart).In your example it will be t1+t2+t3+t4 (if all shards were on that
node). The chart you see then plots the derivative (change per second) of
this value.

Hope this helps,
Boaz

On Wednesday, April 16, 2014 11:25:38 AM UTC+2, sindhu wrote:

Hi,

I'm new to elastic search and marvel . I'm currently investigating marvel
and I want to know if the "Indices total indexing time" shown by marvel is
"sum of the times taken by each shard" or "time between the arrival of
request and completion of indexing".

For example if there are 5 shards and indexing request arrives at 's' and
indexing completes at 'e'. And shards take time t1,t2,t3,t4 and t5
respectively. So how is Indices total indexing time calculated now. Is ut
e-s or t1+t2+t3+t4+t5.

To add to this question, just want to know if marvel uses stats api.

Thanks.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/1pKOwD7y_Bg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8c04d37e-1b69-44de-af7c-ab94ac474cfc%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8c04d37e-1b69-44de-af7c-ab94ac474cfc%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMWeVUvXXKUBdTcNz2W1gGaBhhmYgGjjW2DkaMgJUQrV1C6U-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(sindhu) #4

Hi,

Thanks for the reply. Helped a lot. I also have a query about index rate
total. Is it

  1. no of documents indexed per sec
  2. no of indexing requests coming per sec
  3. no of indexing requests completed per sec

In my case i gave 42 files for indexing

{
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"_all": {
"primaries": {
"docs": {
"count": 48460,
"deleted": 0
},
"store": {
"size_in_bytes": 1089957,
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 42,
"index_time_in_millis": 1645,
"index_current": 0,
"delete_total": 0,
"delete_time_in_millis": 0,
"delete_current": 0
},

So what exactly does the graph of metric "Indexing request Rate" represnts.

Can you also provide any links to documentation about what each metric
exactly represents if possible.

Thanks,
sindhu.

On Thu, Apr 17, 2014 at 6:14 PM, sindhuri nadella <
sindhuri.nadella@gmail.com> wrote:

Hi,

Thanks for the repy. Helped a lot. I also have a query about index rate
total. Is it

  1. no of documents indexed per sec
  2. no of indexing req coming per sec
  3. no of indexing requests completed per sec

Can you also provide any links to documentation about what each metric
exactly represents if possible.

Thanks,
sindhu.

On Thu, Apr 17, 2014 at 1:30 AM, Boaz Leskes b.leskes@gmail.com wrote:

Hi,

By the title, I assume you refer to the chart show in the Node Stats
dashboard. This one is tricky. It shows you the total spent on indexing per
second. This is best explained by an example - say you have 2 shards on a
node. Say that each indexes 100% of the time (continuously indexing) and
they do so in parallel. In this case we have 2 seconds of indexing (total
of two shards) for each real-world second. In other words - see this as a
load factor of indexing on the node.

Technically we indeed use the node stats api which gives you the total
spent on indexing of all shards that ever were on this node (reset on
restart).In your example it will be t1+t2+t3+t4 (if all shards were on that
node). The chart you see then plots the derivative (change per second) of
this value.

Hope this helps,
Boaz

On Wednesday, April 16, 2014 11:25:38 AM UTC+2, sindhu wrote:

Hi,

I'm new to elastic search and marvel . I'm currently investigating
marvel and I want to know if the "Indices total indexing time" shown by
marvel is "sum of the times taken by each shard" or "time between the
arrival of request and completion of indexing".

For example if there are 5 shards and indexing request arrives at 's'
and indexing completes at 'e'. And shards take time t1,t2,t3,t4 and t5
respectively. So how is Indices total indexing time calculated now. Is ut
e-s or t1+t2+t3+t4+t5.

To add to this question, just want to know if marvel uses stats api.

Thanks.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/1pKOwD7y_Bg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8c04d37e-1b69-44de-af7c-ab94ac474cfc%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8c04d37e-1b69-44de-af7c-ab94ac474cfc%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMWeVUt3G5Tqq4ryT%2Bgc7vJmxVDUAt9GiFzkBn%2B-i4ncnF5xGA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(sindhu) #5

Hi,

Thanks for the reply. Helped a lot. I also have a query about index rate
total. Is it

  1. no of documents indexed per sec
  2. no of indexing requests coming per sec
  3. no of indexing requests completed per sec

In my case i gave 42 files for indexing and following is the output of
codeindex_stats where codeindex is the index name.

{
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"_all": {
"primaries": {
"docs": {
"count": 48460,
"deleted": 0
},
"store": {
"size_in_bytes": 1089957,
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 42,
"index_time_in_millis": 1645,
"index_current": 0,
"delete_total": 0,
"delete_time_in_millis": 0,
"delete_current": 0
},

So what exactly does the graph of metric "Indexing request Rate"
represents. How do you calculate the index rate?

Can you also provide any links to documentation about what each metric
exactly represents if possible.

Thanks,
sindhu.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c20d5d6e-0cc9-4213-ad21-ee6830b07259%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Boaz Leskes) #6

The Indexing request Rate chart shows the number of complete indexing doc
operations (index/create) per second. It counts on the document level so a
bulk request with 10 docs is equal to 10 requests with a single doc. The
counter is updated when the operation is completed.

I think the chart title is misleading as it refers to requests and that's
not accurate. I will change it.

On Thursday, April 17, 2014 4:07:30 PM UTC+2, sindhu wrote:

Hi,

Thanks for the reply. Helped a lot. I also have a query about index rate
total. Is it

  1. no of documents indexed per sec
  2. no of indexing requests coming per sec
  3. no of indexing requests completed per sec

In my case i gave 42 files for indexing and following is the output of
codeindex_stats where codeindex is the index name.

{
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"_all": {
"primaries": {
"docs": {
"count": 48460,
"deleted": 0
},
"store": {
"size_in_bytes": 1089957,
"throttle_time_in_millis": 0
},
"indexing": {
"index_total": 42,
"index_time_in_millis": 1645,
"index_current": 0,
"delete_total": 0,
"delete_time_in_millis": 0,
"delete_current": 0
},

So what exactly does the graph of metric "Indexing request Rate"
represents. How do you calculate the index rate?

Can you also provide any links to documentation about what each metric
exactly represents if possible.

Thanks,
sindhu.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9cc610b1-1b19-4138-a4eb-00d585bc0906%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #7