# Marvel shows sporadic indexing activity - is this correct? **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring\ **Created:** [January 6, 2016, 8:33am UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474 "2016-01-06T08:33:13Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![yehosef](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yehosef/32/42175_2.png) [@yehosef](https://discuss.elastic.co/u/yehosef)\ **Post date:** [January 6, 2016, 8:33am UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/1 "2016-01-06T08:33:13Z")
We have an indexing process that is indexing data from 4 servers each with about 10-20 processes. Each one does a bulk upload after it processes enough data. Because there are so many process, I would assume the insertions to be relatively smooth. but in marvel I see frequent periodic spikes in indexing, followed by zero activity, repeated. ![](https://us1.discourse-cdn.com/elastic/original/2X/d/debb0f3e9b9a1f8d8b187862d5ebd2d02a6a0250.png) Is this due to how my index is configured or some issue in marvel? ---
**Author:** ![yehosef](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yehosef/32/42175_2.png) [@yehosef](https://discuss.elastic.co/u/yehosef)\ **Post date:** [January 11, 2016, 8:37am UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/2 "2016-01-11T08:37:28Z")
bump... ---
**Author:** ![tsullivan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tsullivan/32/31077_2.png) [@tsullivan](https://discuss.elastic.co/u/tsullivan)\ **Post date:** [January 11, 2016, 6:02pm UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/3 "2016-01-11T18:02:47Z")
If the data uploading was happening at a constant and consistent rate, the line graph would show much less variation. The `marvel.agent.interval` setting controls how frequently data samples are collected. If it is still at the default setting of 10 seconds, and there are 10-second periods of time where no data has been indexed, that would explain why you have zeros on your Y-axis. I can only think that the your bulk uploads are in sync, and are collectively uploading data about every 37-38 seconds (5 minutes / 8). If you want to smooth out the actual indexing rate, you could chunk up the data before the processes upload it, to break up the data into multiple bulk inserts. If you just would like to see the Marvel chart smoothed out, you could try bumping up the `marvel.agent.interval` setting to 40 seconds. ---
**Author:** ![yehosef](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yehosef/32/42175_2.png) [@yehosef](https://discuss.elastic.co/u/yehosef)\ **Post date:** [January 12, 2016, 8:00am UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/4 "2016-01-12T08:00:42Z")
There are up to 40-60 long-running scripts on 4 machines - each one takes a item from a query and does some queries, processes it, and then does a bulk upload once it reaches an internal queue limit (~1000 documents). It's inconceivable to me that these all "magically" sync up consistently. I took a closer look and other internal metrics such as the latencies also drop to 0 at the same time. ![](https://us1.discourse-cdn.com/elastic/original/2X/4/4fd628a221beda7774fcb31073da09d905494200.png) Is there something else that could have an effect? On other indices I've changed the refresh interval to 30s (I don't think I've changed the .marvel-es - but maybe something like that? If you zoom into to the 5 minute resolution so you see sub-minute resolution are your graphs smooth? Any other ideas? There is definitely something not right (it seems!) ---
**Author:** ![tsullivan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tsullivan/32/31077_2.png) [@tsullivan](https://discuss.elastic.co/u/tsullivan)\ **Post date:** [January 12, 2016, 5:28pm UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/5 "2016-01-12T17:28:41Z")
> If you zoom into to the 5 minute resolution so you see sub-minute resolution are your graphs smooth? If you zoom into a fine-grained resolution, the graphs are not expected to be smooth because the metrics are queried using a derivative aggregation, where each data point represents a bucket of time. If there are no documents within that time bucket, then the line chart will go to 0 for that time bucket. Reference: [Derivative aggregation | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-derivative-aggregation.html) See also: [Marvel 2.0 Uneven measurement times result in odd graphs](https://discuss.elastic.co/t/marvel-2-0-uneven-measurement-times-result-in-odd-graphs/34841) I set up a test where I index 1 document roughly every 15 seconds. On the Indices overview, when my time range is 30 minutes, my charts look like what I would expect: ![](https://us1.discourse-cdn.com/elastic/original/2X/6/68d5844f8c987ffff29c32ffda803ad5b5b54a73.png) When I make the time range 15 minutes, the graphs show zeroes because some of the time buckets shown have 0 doc count - there was no data available to calculate the derivative: ![](https://us1.discourse-cdn.com/elastic/original/2X/6/6e1ac4fd60b1cd62fe1078a445c5d7d60ae092f0.png) The data sometimes go to zero in my data, but only for non-consecutive data points. ---
**Author:** ![yehosef](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yehosef/32/42175_2.png) [@yehosef](https://discuss.elastic.co/u/yehosef)\ **Post date:** [January 13, 2016, 9:24am UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/6 "2016-01-13T09:24:24Z")
Just to be clear - I'm indexing all the time - I'm currently running a reindex where I have 80 scripts running on 4 servers sending bulk requests. There is no way this graph represents reality. ![](https://us1.discourse-cdn.com/elastic/original/2X/c/ca31ae40f910f633e1c7b59926ad6594620db0d6.png) > [@tsullivan](#): > > If you zoom into a fine-grained resolution, the graphs are not expected to be smooth because the metrics are queried using a derivative aggregation, where each data point represents a bucket of time. If there are no documents within that time bucket, then the line chart will go to 0 for that time bucket. Then I would consider this as a bug in Marvel. Marvel should either: 1. make sure to collect data every second so there is no missing data (normally - there will always be exceptions - but in this case it's the norm - every minute it misses data) 2. not return data for these points - and the graphing package should skip the nulls 3. apply a moving average to smooth out the missing data so the differential doesn't break. 2 and 3 are still problematic because it basically means the data is broken - if I have 4000 rq/s for 4 out of 6 data points when I zoom out I'll see the line as the average which is around 2600 (am I mistaken here?). even when zooming out - it shows jumpiness which is not real. The real problem is 1 - which I assume is caused by a bug or misconfiguration in marvel-agent to not collect the data properly. In the chart above I'm collecting only 1 data point per minute. ---
**Author:** ![tsullivan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tsullivan/32/31077_2.png) [@tsullivan](https://discuss.elastic.co/u/tsullivan)\ **Post date:** [January 14, 2016, 1:00am UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/7 "2016-01-14T01:00:48Z")
You could have your marvel agent collect statistics every second, by setting the `marvel.agent.interval` option to `1s` in elasticsearch.yml. All the options available to configure the marvel agent are here: [https://www.elastic.co/guide/en/marvel/current/configuration.html](https://www.elastic.co/guide/en/marvel/current/configuration.html) ---
**Author:** ![tanguy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tanguy/32/6030_2.png) [@tanguy](https://discuss.elastic.co/u/tanguy)\ **Post date:** [January 14, 2016, 8:08am UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/8 "2016-01-14T08:08:11Z")
> [@tsullivan](#): > > You could have your marvel agent collect statistics every second, by setting the marvel.agent.interval option to 1s in elasticsearch.yml. Be careful when setting the interval to a low value - depending of your cluster it can take more than 1 second to grab all the metrics. ---
**Author:** ![yehosef](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yehosef/32/42175_2.png) [@yehosef](https://discuss.elastic.co/u/yehosef)\ **Post date:** [January 14, 2016, 10:09am UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/9 "2016-01-14T10:09:03Z")
I said every second - but what I really meant was "every time it's supposed to" - This is set to collect every 10 seconds - which is fine - but that fact that it's missing data is the problem. Is the `gap_policy` for these aggregations set to `skip` ? If not, perhaps that would avoid part of the problem. > [@tanguy](#): > > Be careful when setting the interval to a low value - depending of your cluster it can take more than 1 second to grab all the metrics. perhaps this is part of the problem - maybe 10 seconds is too fast, especially when the server is under load and it's dropping data? I'll try a higher value. ---
**Author:** ![tanguy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tanguy/32/6030_2.png) [@tanguy](https://discuss.elastic.co/u/tanguy)\ **Post date:** [March 2, 2016, 3:22pm UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/10 "2016-03-02T15:22:03Z")
> [@yehosef](#): > > perhaps this is part of the problem - maybe 10 seconds is too fast, especially when the server is under load and it's dropping data? I'll try a higher value. That's what I suspect here. Do you still have this issue? If so, it would be nice to have the output of `/_nodes/hot_threads` and `/_cluster/pending_tasks` during the "peak" (you may need to run it multiple times). ---
**Author:** ![yehosef](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yehosef/32/42175_2.png) [@yehosef](https://discuss.elastic.co/u/yehosef)\ **Post date:** [March 7, 2016, 10:57am UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/11 "2016-03-07T10:57:49Z")
It's not a peak issue.. it happens all the time. ![](https://us1.discourse-cdn.com/elastic/original/2X/e/e834d019caa07160d7aad77210429124ff4cacf6.png) I've pretty much given up on marvel. We still use it but it's really not as great of a tool as it was. While I understand the move to kibana 4 for the elastic ecosystem - it's a huge step backwards in practical usefulness. I haven't had time to install something else - but I'll probably just use a es-graphite/influx stats tool with a grafana dashboard. ---
**Author:** ![skearns](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/skearns/32/125945_2.png) [@skearns](https://discuss.elastic.co/u/skearns)\ **Post date:** [March 7, 2016, 5:23pm UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/12 "2016-03-07T17:23:35Z")
Hi Yehosef, I appreciate your frustration - we are continuing to improve Marvel with each release. I didn't see in this issue which version you are running, but we have dramatically improved things since the 2.0 release, and the upcoming 2.3 release makes a number of other big improvements (e.g. giving you flexibility in how you define node uniqueness, and big improvements to the underlying datamodel that we capture, opening the door for more rapidly building new features). We're totally committed to making Marvel the best way to monitor the entire Elastic Stack, and we appreciate your patience (and your feedback!) as we continue to improve. Please don't hesitate to open other threads if you run into other issues. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:45pm UTC](https://discuss.elastic.co/t/marvel-shows-sporadic-indexing-activity-is-this-correct/38474/13 "2017-07-06T13:45:02Z")