# Masters are ramping up GC times, using more heap every minute

**URL:** <https://discuss.elastic.co/t/masters-are-ramping-up-gc-times-using-more-heap-every-minute/130275>\
**Category:** Elasticsearch\
**Created:** [May 2, 2018, 2:54pm UTC](https://discuss.elastic.co/t/masters-are-ramping-up-gc-times-using-more-heap-every-minute/130275 "2018-05-02T14:54:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![xrl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xrl/32/3556_2.png) [@xrl](https://discuss.elastic.co/u/xrl)\
**Post date:** [May 2, 2018, 2:54pm UTC](https://discuss.elastic.co/t/masters-are-ramping-up-gc-times-using-more-heap-every-minute/130275/1 "2018-05-02T14:54:16Z")

</div>

I am running a cluster with 20 9TB data nodes (32GB heap), 3 client nodes (24GB heap), and 3 master nodes (20GB heap). This cluster has been running stable under monitoring for the past 90 days. Yesterday, nowhere near a UTC rollover event which would create new daily indices, we started seeing ramped up GC time on the masters. The GCs are more frequent and taking longer, here's a graph:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/2/129faf5e116c8b1582720d9520fb49f552a4403b.png)

The two saw-tooths are from my manual restart of the master. The newly elected masters show the same memory growth.

Here is the time spent in GC:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/e/ae979f3a01750a05dec9275bde28f650d0ff850e.png)

What can I do to debug this issue? In the past it has been a problem with out-of-control schema growth from poorly processed data. But here I'm not sure, how can I track down how memory is being used in the master?

---

<div class="post-metadata">

**Author:** ![xrl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xrl/32/3556_2.png) [@xrl](https://discuss.elastic.co/u/xrl)\
**Post date:** [May 2, 2018, 7:53pm UTC](https://discuss.elastic.co/t/masters-are-ramping-up-gc-times-using-more-heap-every-minute/130275/2 "2018-05-02T19:53:18Z")

</div>

Turns out one of our analysts was logging bro data directly to the cluster and had turned it on right at the time we saw this issue start. Disabling their data flow got the master under control again.

We are still trying to figure out what is causing the bro IDS data to induce master GC pressure.

---

<div class="post-metadata">

**Author:** ![Glen\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glen_smith/32/111656_2.png) [@Glen\_Smith](https://discuss.elastic.co/u/Glen_Smith)\
**Post date:** [May 5, 2018, 4:49am UTC](https://discuss.elastic.co/t/masters-are-ramping-up-gc-times-using-more-heap-every-minute/130275/3 "2018-05-05T04:49:51Z")

</div>

I've seen this kind of thing when there is a field explosion due to inadvisably structured documents being indexed, i.e. something that should be a field _value_ is in the documents as a field _name_, causing a high rate of dynamic addition of fields to your mapping. A particular tell for this (in addition to looking at the mapping and seeing thousands of fields) is that heap usage is extraordinarily high even on the unelected masters.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 2, 2018, 4:49am UTC](https://discuss.elastic.co/t/masters-are-ramping-up-gc-times-using-more-heap-every-minute/130275/4 "2018-06-02T04:49:52Z")

</div>

Closed on June 2, 2018, 4:49am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.