# Apparent memory leak after a few days of heavy indexing
**URL:** https://discuss.elastic.co/t/apparent-memory-leak-after-a-few-days-of-heavy-indexing/12834
**Category:** Elasticsearch
**Created:** [July 17, 2013, 11:57pm UTC](https://discuss.elastic.co/t/apparent-memory-leak-after-a-few-days-of-heavy-indexing/12834 "2013-07-17T23:57:02Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@rafe](https://discuss.elastic.co/u/rafe)
#### Post date: [July 17, 2013, 11:57pm UTC](https://discuss.elastic.co/t/apparent-memory-leak-after-a-few-days-of-heavy-indexing/12834/1 "2013-07-17T23:57:02Z")
I have a two node cluster. Each node runs with a heap size 16GB. The nodes
run on their own boxes and have plenty of CPU to themselves. Currently, the
cluster's only workload is indexing at pretty high volume. We are indexing
using the bulk index API, and are sending about 10 batches of 400 documents
per second. We're using the Java client, specifically TransportClient.
Things work well for a little while (1-2 days), but eventually, the cluster
falls over -- see the heap usage chart from Graphite. This is for just one
host, but the memory behavior is identical across the two nodes.
[image: Inline image 1]
This looks like a memory leak to me. Logs don't reveal anything out of the
ordinary happened when heap usage started increasing linearly. A few common
sources of memory issues that I've already ruled out:
- The cache. Since the workload is basically entirely indexing, the
filter and field caches shouldn't even come into play here. Either way,
both are configured to be limited in size, so this can't be it.
- "Not enough heap" -- I can reproduce this no matter how much heap
space I give my nodes.
Anyone seen this before, or any ideas as to what my issue might be here?
I've turned on TRACE logging for Lucene merges, but I don't think I'll get
any good data out of that until my cluster crashes again.
Rafe
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [July 18, 2013, 7:03am UTC](https://discuss.elastic.co/t/apparent-memory-leak-after-a-few-days-of-heavy-indexing/12834/2 "2013-07-18T07:03:37Z")
You don't tell us the ES version and the JVM heap memory you have
configured, so I assume standard settings? (Image is not viewable).
I recommend increasing the heap, and also the following settings for
smoother indexing (less peak resource intensive)
index.merge.policy.max\_merged\_segment: 2g
index.merge.policy.segments\_per\_tier: 24
index.merge.policy.max\_merge\_at\_once: 8
And for better fail safety, consider a 3 node cluster.
Jörg
Am 18.07.13 01:57, schrieb [rafe@squareup.com](mailto:rafe@squareup.com):
> I have a two node cluster. Each node runs with a heap size 16GB. The
> nodes run on their own boxes and have plenty of CPU to themselves.
> Currently, the cluster's only workload is indexing at pretty high
> volume. We are indexing using the bulk index API, and are sending
> about 10 batches of 400 documents per second. We're using the Java
> client, specifically TransportClient.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [July 18, 2013, 9:38am UTC](https://discuss.elastic.co/t/apparent-memory-leak-after-a-few-days-of-heavy-indexing/12834/3 "2013-07-18T09:38:23Z")
Can you open an issue and share an example so we can recreate it and try
and chase it up?
On Thu, Jul 18, 2013 at 1:57 AM, [rafe@squareup.com](mailto:rafe@squareup.com) wrote:
> I have a two node cluster. Each node runs with a heap size 16GB. The nodes
> run on their own boxes and have plenty of CPU to themselves. Currently, the
> cluster's only workload is indexing at pretty high volume. We are indexing
> using the bulk index API, and are sending about 10 batches of 400 documents
> per second. We're using the Java client, specifically TransportClient.
>
> Things work well for a little while (1-2 days), but eventually, the
> cluster falls over -- see the heap usage chart from Graphite. This is for
> just one host, but the memory behavior is identical across the two nodes.
>
> [image: Inline image 1]
>
> This looks like a memory leak to me. Logs don't reveal anything out of the
> ordinary happened when heap usage started increasing linearly. A few common
> sources of memory issues that I've already ruled out:
>
> - The cache. Since the workload is basically entirely indexing, the
> filter and field caches shouldn't even come into play here. Either way,
> both are configured to be limited in size, so this can't be it.
> - "Not enough heap" -- I can reproduce this no matter how much heap
> space I give my nodes.
>
> Anyone seen this before, or any ideas as to what my issue might be here?
> I've turned on TRACE logging for Lucene merges, but I don't think I'll get
> any good data out of that until my cluster crashes again.
>
> Rafe
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@rafe](https://discuss.elastic.co/u/rafe)
#### Post date: [July 18, 2013, 4:32pm UTC](https://discuss.elastic.co/t/apparent-memory-leak-after-a-few-days-of-heavy-indexing/12834/4 "2013-07-18T16:32:24Z")
Sorry for the incomplete information. I am using ES 0.90.1 and I am using
default JVM memory settings, with the exception of -Xmx16g and -Xms16g. The
chart I tried to attach before is at [http://i.imgur.com/vYeDEsc.png](http://i.imgur.com/vYeDEsc.png),
hopefully the link I gave won't be stripped out as well.
Jörg, could merging cause the types of pauses and memory usage I am seeing?
A few hours and like 6GB seems unreasonable for a merge, and also seems
really unlikely based on my experience with Lucene.
Shay, I will work on submitting a ticket.
Rafe
On Thu, Jul 18, 2013 at 12:03 AM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:
> You don't tell us the ES version and the JVM heap memory you have
> configured, so I assume standard settings? (Image is not viewable).
>
> I recommend increasing the heap, and also the following settings for
> smoother indexing (less peak resource intensive)
>
> index.merge.policy.max\_merged\_\*\*segment: 2g
> index.merge.policy.segments\_\*\*per\_tier: 24
> index.merge.policy.max\_merge\_\*\*at\_once: 8
>
> And for better fail safety, consider a 3 node cluster.
>
> Jörg
>
> Am 18.07.13 01:57, schrieb [rafe@squareup.com](mailto:rafe@squareup.com):
>
> I have a two node cluster. Each node runs with a heap size 16GB. The
>
> > nodes run on their own boxes and have plenty of CPU to themselves.
> > Currently, the cluster's only workload is indexing at pretty high volume.
> > We are indexing using the bulk index API, and are sending about 10 batches
> > of 400 documents per second. We're using the Java client, specifically
> > TransportClient.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\*
> topic/elasticsearch/cS-\*\*XVa6jEmU/unsubscribe[https://groups.google.com/d/topic/elasticsearch/cS-XVa6jEmU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cS-XVa6jEmU/unsubscribe)
> .
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com)
> .
> For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> .
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@vinh](https://discuss.elastic.co/u/vinh)
#### Post date: [July 18, 2013, 4:48pm UTC](https://discuss.elastic.co/t/apparent-memory-leak-after-a-few-days-of-heavy-indexing/12834/5 "2013-07-18T16:48:48Z")
Rafe, just curious…what size are your indexes? Do you cap them and create a new index once the size is reached? Or are you indexing into a single index?
-Vinh
On Jul 18, 2013, at 9:32 AM, Rafe Kettler [rafe@squareup.com](mailto:rafe@squareup.com) wrote:
> Sorry for the incomplete information. I am using ES 0.90.1 and I am using default JVM memory settings, with the exception of -Xmx16g and -Xms16g. The chart I tried to attach before is at [http://i.imgur.com/vYeDEsc.png](http://i.imgur.com/vYeDEsc.png), hopefully the link I gave won't be stripped out as well.
>
> Jörg, could merging cause the types of pauses and memory usage I am seeing? A few hours and like 6GB seems unreasonable for a merge, and also seems really unlikely based on my experience with Lucene.
>
> Shay, I will work on submitting a ticket.
>
> Rafe
>
> On Thu, Jul 18, 2013 at 12:03 AM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:
> You don't tell us the ES version and the JVM heap memory you have configured, so I assume standard settings? (Image is not viewable).
>
> I recommend increasing the heap, and also the following settings for smoother indexing (less peak resource intensive)
>
> index.merge.policy.max\_merged\_segment: 2g
> index.merge.policy.segments\_per\_tier: 24
> index.merge.policy.max\_merge\_at\_once: 8
>
> And for better fail safety, consider a 3 node cluster.
>
> Jörg
>
> Am 18.07.13 01:57, schrieb [rafe@squareup.com](mailto:rafe@squareup.com):
>
> I have a two node cluster. Each node runs with a heap size 16GB. The nodes run on their own boxes and have plenty of CPU to themselves. Currently, the cluster's only workload is indexing at pretty high volume. We are indexing using the bulk index API, and are sending about 10 batches of 400 documents per second. We're using the Java client, specifically TransportClient.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/cS-XVa6jEmU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cS-XVa6jEmU/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@rafe](https://discuss.elastic.co/u/rafe)
#### Post date: [July 18, 2013, 4:57pm UTC](https://discuss.elastic.co/t/apparent-memory-leak-after-a-few-days-of-heavy-indexing/12834/6 "2013-07-18T16:57:22Z")
Vinh, I am creating one index per day's worth of data (so there is no size
threshold for creating a new index). My indices are ~500 million docs, 75GB
on disk (for the primary shards).
Rafe
On Thu, Jul 18, 2013 at 9:48 AM, vinh [vinh@loggly.com](mailto:vinh@loggly.com) wrote:
> Rafe, just curious…what size are your indexes? Do you cap them and create
> a new index once the size is reached? Or are you indexing into a single
> index?
> -Vinh
>
> On Jul 18, 2013, at 9:32 AM, Rafe Kettler [rafe@squareup.com](mailto:rafe@squareup.com) wrote:
>
> Sorry for the incomplete information. I am using ES 0.90.1 and I am using
> default JVM memory settings, with the exception of -Xmx16g and -Xms16g. The
> chart I tried to attach before is at [http://i.imgur.com/vYeDEsc.png](http://i.imgur.com/vYeDEsc.png),
> hopefully the link I gave won't be stripped out as well.
>
> Jörg, could merging cause the types of pauses and memory usage I am
> seeing? A few hours and like 6GB seems unreasonable for a merge, and also
> seems really unlikely based on my experience with Lucene.
>
> Shay, I will work on submitting a ticket.
>
> Rafe
>
> On Thu, Jul 18, 2013 at 12:03 AM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com)wrote:
>
> > You don't tell us the ES version and the JVM heap memory you have
> > configured, so I assume standard settings? (Image is not viewable).
> >
> > I recommend increasing the heap, and also the following settings for
> > smoother indexing (less peak resource intensive)
> >
> > index.merge.policy.max\_merged\_\*\*segment: 2g
> > index.merge.policy.segments\_\*\*per\_tier: 24
> > index.merge.policy.max\_merge\_\*\*at\_once: 8
> >
> > And for better fail safety, consider a 3 node cluster.
> >
> > Jörg
> >
> > Am 18.07.13 01:57, schrieb [rafe@squareup.com](mailto:rafe@squareup.com):
> >
> > I have a two node cluster. Each node runs with a heap size 16GB. The
> >
> > > nodes run on their own boxes and have plenty of CPU to themselves.
> > > Currently, the cluster's only workload is indexing at pretty high volume.
> > > We are indexing using the bulk index API, and are sending about 10 batches
> > > of 400 documents per second. We're using the Java client, specifically
> > > TransportClient.
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "elasticsearch" group.
> > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\*
> > topic/elasticsearch/cS-\*\*XVa6jEmU/unsubscribe[https://groups.google.com/d/topic/elasticsearch/cS-XVa6jEmU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cS-XVa6jEmU/unsubscribe)
> > .
> > To unsubscribe from this group and all its topics, send an email to
> > elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com)
> > .
> > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > .
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
>
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> [https://groups.google.com/d/topic/elasticsearch/cS-XVa6jEmU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/cS-XVa6jEmU/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:25am UTC](https://discuss.elastic.co/t/apparent-memory-leak-after-a-few-days-of-heavy-indexing/12834/7 "2017-07-06T02:25:41Z")