# Caches and Garbage collection
**URL:** https://discuss.elastic.co/t/caches-and-garbage-collection/11528
**Category:** Elasticsearch
**Created:** [April 10, 2013, 12:36pm UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528 "2013-04-10T12:36:04Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)
#### Post date: [April 10, 2013, 12:36pm UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528/1 "2013-04-10T12:36:04Z")
Hi,
In the beginning itself, I would like to say that the problem can be solved
by adding more nodes or increasing RAM but I can't do that right now.
My problem is, every two or three days, I've to execute "curl -XPOST '
[http://localhost:9200/\_cache/clear](http://localhost:9200/_cache/clear)'" to decrease the heap usage. This
starts happening after the indices size grow a certain limit, say 3TB.
Couple of questions that I have:
1. Although I've set "index.cache.field.type: soft", GC is not able to
reduce heap usage.
2. When I execute that command to clear caches, the indexing stops for a
couple of minutes. Why is that?
3. I don't search for documents on all indices. I only use the latest 7
indices or so. Is there a way that I can disable caching for older indices?
Clearing caches decreases search performance but I can live with it. Is
there a way I can optimize RAM usage by compromising search speed? I just
want to limit the size of cache ES uses.
--
Regards,
Abhijeet Rastogi (shadyabhi)
> **[Abhijeet's Blog](https://blog.abhijeetr.com/)**
>
> I write about problems that I solve
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [April 10, 2013, 5:13pm UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528/2 "2013-04-10T17:13:22Z")
If you are not searching older indices you can just close them, which will
release cache for these indices. Alternatively, you can clear cache only
for indices that you no longer use. Cleaning cache on all indices makes
little sense because under heavy search traffic elasticsearch will rebuild
all these caches right away causing very sluggish performance in the
process. And, by the way, using soft cache is like playing russian roulette
- you basically just betting on not having a diverse enough set of queries
that will load all values into the cache at the same time and bring your
server down.
On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:
> Hi,
>
> In the beginning itself, I would like to say that the problem can be
> solved by adding more nodes or increasing RAM but I can't do that right
> now.
>
> My problem is, every two or three days, I've to execute "curl -XPOST '
> [http://localhost:9200/\_cache/clear](http://localhost:9200/_cache/clear)'" to decrease the heap usage. This
> starts happening after the indices size grow a certain limit, say 3TB.
>
> Couple of questions that I have:
>
> 1. Although I've set "index.cache.field.type: soft", GC is not able to
> reduce heap usage.
> 2. When I execute that command to clear caches, the indexing stops for a
> couple of minutes. Why is that?
> 3. I don't search for documents on all indices. I only use the latest 7
> indices or so. Is there a way that I can disable caching for older indices?
>
> Clearing caches decreases search performance but I can live with it. Is
> there a way I can optimize RAM usage by compromising search speed? I just
> want to limit the size of cache ES uses.
>
> --
> Regards,
> Abhijeet Rastogi (shadyabhi)
> [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)
#### Post date: [April 10, 2013, 5:19pm UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528/3 "2013-04-10T17:19:58Z")
Hi Igor,
Thanks for the reply. Can you explain the pauses in indexing when I clear
cache?
Can you elaborate the part you said about soft cache, I didn't get what you
meant. Also, why isn't GC clearing the cache even though it's soft?
On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote:
> If you are not searching older indices you can just close them, which will
> release cache for these indices. Alternatively, you can clear cache only
> for indices that you no longer use. Cleaning cache on all indices makes
> little sense because under heavy search traffic elasticsearch will rebuild
> all these caches right away causing very sluggish performance in the
> process. And, by the way, using soft cache is like playing russian roulette
>
> - you basically just betting on not having a diverse enough set of queries
> that will load all values into the cache at the same time and bring your
> server down.
>
> On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:
>
> > Hi,
> >
> > In the beginning itself, I would like to say that the problem can be
> > solved by adding more nodes or increasing RAM but I can't do that right
> > now.
> >
> > My problem is, every two or three days, I've to execute "curl -XPOST '
> > [http://localhost:9200/\_cache/\*\*clear](http://localhost:9200/_cache/**clear) [http://localhost:9200/\_cache/clear](http://localhost:9200/_cache/clear)'"
> > to decrease the heap usage. This starts happening after the indices size
> > grow a certain limit, say 3TB.
> >
> > Couple of questions that I have:
> >
> > 1. Although I've set "index.cache.field.type: soft", GC is not able to
> > reduce heap usage.
> > 2. When I execute that command to clear caches, the indexing stops for a
> > couple of minutes. Why is that?
> > 3. I don't search for documents on all indices. I only use the latest 7
> > indices or so. Is there a way that I can disable caching for older indices?
> >
> > Clearing caches decreases search performance but I can live with it. Is
> > there a way I can optimize RAM usage by compromising search speed? I just
> > want to limit the size of cache ES uses.
> >
> > --
> > Regards,
> > Abhijeet Rastogi (shadyabhi)
> > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
Regards,
Abhijeet Rastogi (shadyabhi)
[http://blog.abhijeetr.com](http://blog.abhijeetr.com)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [April 10, 2013, 5:42pm UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528/4 "2013-04-10T17:42:09Z")
I would guess indexing pauses are caused by I/O and CPU contention. There
are no locks that I am aware of that would prevent indexing from going on,
but by cleaning cache you suddenly force Elasticsearch to reload Gigabytes
of data from disk and rebuild all these caches in memory and it takes toll
on all other processes.
In order to perform a query that uses a field cache elasticsearch needs to
load entire field cache into memory. So, if you end up with several queries
each trying to load field cache for their set of fields, you might create a
situation when queries are fighting each other over limited memory evicting
each other's cache and trying to load it over and over again.
Soft cache is cleaned only when it's not used and there is GC pressure.
It's possible that you never achieve these condition. I also suspect
that eviction counter might not be reporting evictions correctly in case of
soft cache.
On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:
> Hi Igor,
>
> Thanks for the reply. Can you explain the pauses in indexing when I clear
> cache?
>
> Can you elaborate the part you said about soft cache, I didn't get what
> you meant. Also, why isn't GC clearing the cache even though it's soft?
>
> On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov \<[imo...@gmail.com](mailto:imo...@gmail.com)\
>
> > wrote:
>
> > If you are not searching older indices you can just close them, which
> > will release cache for these indices. Alternatively, you can clear cache
> > only for indices that you no longer use. Cleaning cache on all indices
> > makes little sense because under heavy search traffic elasticsearch will
> > rebuild all these caches right away causing very sluggish performance in
> > the process. And, by the way, using soft cache is like playing
> > russian roulette - you basically just betting on not having a diverse
> > enough set of queries that will load all values into the cache at the same
> > time and bring your server down.
> >
> > On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:
> >
> > > Hi,
> > >
> > > In the beginning itself, I would like to say that the problem can be
> > > solved by adding more nodes or increasing RAM but I can't do that right
> > > now.
> > >
> > > My problem is, every two or three days, I've to execute "curl -XPOST '
> > > [http://localhost:9200/\_cache/\*\*clear](http://localhost:9200/_cache/**clear)[http://localhost:9200/\_cache/clear](http://localhost:9200/_cache/clear)'"
> > > to decrease the heap usage. This starts happening after the indices size
> > > grow a certain limit, say 3TB.
> > >
> > > Couple of questions that I have:
> > >
> > > 1. Although I've set "index.cache.field.type: soft", GC is not able to
> > > reduce heap usage.
> > > 2. When I execute that command to clear caches, the indexing stops for a
> > > couple of minutes. Why is that?
> > > 3. I don't search for documents on all indices. I only use the latest 7
> > > indices or so. Is there a way that I can disable caching for older indices?
> > >
> > > Clearing caches decreases search performance but I can live with it. Is
> > > there a way I can optimize RAM usage by compromising search speed? I just
> > > want to limit the size of cache ES uses.
> > >
> > > --
> > > Regards,
> > > Abhijeet Rastogi (shadyabhi)
> > > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> Regards,
> Abhijeet Rastogi (shadyabhi)
> [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)
#### Post date: [April 11, 2013, 3:28pm UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528/5 "2013-04-11T15:28:43Z")
HI Igor,
So, I tested that only if I cleared the latest cache, I got issues of
indexing being paused or very slow. used hot\_threads API and it gave me
this.
87.8% (878.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#45]'
87.5% (874.5ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#38]'
86.8% (867.7ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#44]'
86.3% (863ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#46]'
9.8% (97.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1373]'
6.1% (61.4ms out of 1s) cpu usage by thread
'elasticsearch[index12][refresh][T#15]'
5.9% (58.8ms out of 1s) cpu usage by thread
'elasticsearch[index12][refresh][T#10]'
4.8% (48.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1375]'
0.5% (4.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][scheduler][T#1]'
0.4% (3.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][merge][T#6]'
94.0% (939.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#22]'
92.3% (922.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#23]'
36.8% (367.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#24]'
7.1% (71.3ms out of 1s) cpu usage by thread
'elasticsearch[index11][refresh][T#7]'
7.0% (70.3ms out of 1s) cpu usage by thread
'elasticsearch[index11][refresh][T#6]'
4.1% (40.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1400]'
3.5% (35.2ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1421]'
3.4% (33.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1418]'
2.9% (29.4ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1407]'
2.9% (28.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1415]'
What I don't understand is, even though there is no search activity, ES
just starts building cache right away at that very instant of clearing
cache. I can see bulk threads so indexing is going on, just that it's very
slow.
On Wed, Apr 10, 2013 at 11:12 PM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote:
> I would guess indexing pauses are caused by I/O and CPU contention. There
> are no locks that I am aware of that would prevent indexing from going on,
> but by cleaning cache you suddenly force Elasticsearch to reload Gigabytes
> of data from disk and rebuild all these caches in memory and it takes toll
> on all other processes.
>
> In order to perform a query that uses a field cache elasticsearch needs to
> load entire field cache into memory. So, if you end up with several queries
> each trying to load field cache for their set of fields, you might create a
> situation when queries are fighting each other over limited memory evicting
> each other's cache and trying to load it over and over again.
>
> Soft cache is cleaned only when it's not used and there is GC pressure.
> It's possible that you never achieve these condition. I also suspect
> that eviction counter might not be reporting evictions correctly in case of
> soft cache.
>
> On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:
>
> > Hi Igor,
> >
> > Thanks for the reply. Can you explain the pauses in indexing when I clear
> > cache?
> >
> > Can you elaborate the part you said about soft cache, I didn't get what
> > you meant. Also, why isn't GC clearing the cache even though it's soft?
> >
> > On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov [imo...@gmail.com](mailto:imo...@gmail.com) wrote:
> >
> > > If you are not searching older indices you can just close them, which
> > > will release cache for these indices. Alternatively, you can clear cache
> > > only for indices that you no longer use. Cleaning cache on all indices
> > > makes little sense because under heavy search traffic elasticsearch will
> > > rebuild all these caches right away causing very sluggish performance in
> > > the process. And, by the way, using soft cache is like playing
> > > russian roulette - you basically just betting on not having a diverse
> > > enough set of queries that will load all values into the cache at the same
> > > time and bring your server down.
> > >
> > > On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:
> > >
> > > > Hi,
> > > >
> > > > In the beginning itself, I would like to say that the problem can be
> > > > solved by adding more nodes or increasing RAM but I can't do that right
> > > > now.
> > > >
> > > > My problem is, every two or three days, I've to execute "curl -XPOST '
> > > > [http://localhost:9200/\_cache/\*\*\*\*clear](http://localhost:9200/_cache/ **** clear)[http://localhost:9200/\_cache/clear](http://localhost:9200/_cache/clear)'"
> > > > to decrease the heap usage. This starts happening after the indices size
> > > > grow a certain limit, say 3TB.
> > > >
> > > > Couple of questions that I have:
> > > >
> > > > 1. Although I've set "index.cache.field.type: soft", GC is not able to
> > > > reduce heap usage.
> > > > 2. When I execute that command to clear caches, the indexing stops for
> > > > a couple of minutes. Why is that?
> > > > 3. I don't search for documents on all indices. I only use the latest 7
> > > > indices or so. Is there a way that I can disable caching for older indices?
> > > >
> > > > Clearing caches decreases search performance but I can live with it. Is
> > > > there a way I can optimize RAM usage by compromising search speed? I just
> > > > want to limit the size of cache ES uses.
> > > >
> > > > --
> > > > Regards,
> > > > Abhijeet Rastogi (shadyabhi)
> > > > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > >
> > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > .
> >
> > --
> > Regards,
> > Abhijeet Rastogi (shadyabhi)
> > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
Regards,
Abhijeet Rastogi (shadyabhi)
[http://blog.abhijeetr.com](http://blog.abhijeetr.com)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [April 12, 2013, 5:27pm UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528/6 "2013-04-12T17:27:31Z")
Are you using warmers?
On Thursday, April 11, 2013 11:28:43 AM UTC-4, Abhijeet Rastogi wrote:
> HI Igor,
>
> So, I tested that only if I cleared the latest cache, I got issues of
> indexing being paused or very slow. used hot\_threads API and it gave me
> this.
>
> 87.8% (878.2ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][cache][T#45]'
> 87.5% (874.5ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][cache][T#38]'
> 86.8% (867.7ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][cache][T#44]'
> 86.3% (863ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][cache][T#46]'
> 9.8% (97.9ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][bulk][T#1373]'
> 6.1% (61.4ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][refresh][T#15]'
> 5.9% (58.8ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][refresh][T#10]'
> 4.8% (48.2ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][bulk][T#1375]'
> 0.5% (4.9ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][scheduler][T#1]'
> 0.4% (3.9ms out of 1s) cpu usage by thread
> 'elasticsearch[index12][merge][T#6]'
> 94.0% (939.9ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][cache][T#22]'
> 92.3% (922.8ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][cache][T#23]'
> 36.8% (367.5ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][cache][T#24]'
> 7.1% (71.3ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][refresh][T#7]'
> 7.0% (70.3ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][refresh][T#6]'
> 4.1% (40.8ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][bulk][T#1400]'
> 3.5% (35.2ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][bulk][T#1421]'
> 3.4% (33.9ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][bulk][T#1418]'
> 2.9% (29.4ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][bulk][T#1407]'
> 2.9% (28.5ms out of 1s) cpu usage by thread
> 'elasticsearch[index11][bulk][T#1415]'
>
> What I don't understand is, even though there is no search activity, ES
> just starts building cache right away at that very instant of clearing
> cache. I can see bulk threads so indexing is going on, just that it's very
> slow.
>
> On Wed, Apr 10, 2013 at 11:12 PM, Igor Motov \<[imo...@gmail.com](mailto:imo...@gmail.com)\
>
> > wrote:
>
> > I would guess indexing pauses are caused by I/O and CPU contention. There
> > are no locks that I am aware of that would prevent indexing from going on,
> > but by cleaning cache you suddenly force Elasticsearch to reload Gigabytes
> > of data from disk and rebuild all these caches in memory and it takes toll
> > on all other processes.
> >
> > In order to perform a query that uses a field cache elasticsearch needs
> > to load entire field cache into memory. So, if you end up with several
> > queries each trying to load field cache for their set of fields, you might
> > create a situation when queries are fighting each other over limited memory
> > evicting each other's cache and trying to load it over and over again.
> >
> > Soft cache is cleaned only when it's not used and there is GC pressure.
> > It's possible that you never achieve these condition. I also suspect
> > that eviction counter might not be reporting evictions correctly in case of
> > soft cache.
> >
> > On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:
> >
> > > Hi Igor,
> > >
> > > Thanks for the reply. Can you explain the pauses in indexing when I
> > > clear cache?
> > >
> > > Can you elaborate the part you said about soft cache, I didn't get what
> > > you meant. Also, why isn't GC clearing the cache even though it's soft?
> > >
> > > On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov [imo...@gmail.com](mailto:imo...@gmail.com) wrote:
> > >
> > > > If you are not searching older indices you can just close them, which
> > > > will release cache for these indices. Alternatively, you can clear cache
> > > > only for indices that you no longer use. Cleaning cache on all indices
> > > > makes little sense because under heavy search traffic elasticsearch will
> > > > rebuild all these caches right away causing very sluggish performance in
> > > > the process. And, by the way, using soft cache is like playing
> > > > russian roulette - you basically just betting on not having a diverse
> > > > enough set of queries that will load all values into the cache at the same
> > > > time and bring your server down.
> > > >
> > > > On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > In the beginning itself, I would like to say that the problem can be
> > > > > solved by adding more nodes or increasing RAM but I can't do that right
> > > > > now.
> > > > >
> > > > > My problem is, every two or three days, I've to execute "curl -XPOST '
> > > > > [http://localhost:9200/\_cache/\*\*\*\*clear](http://localhost:9200/_cache/ **** clear)[http://localhost:9200/\_cache/clear](http://localhost:9200/_cache/clear)'"
> > > > > to decrease the heap usage. This starts happening after the indices size
> > > > > grow a certain limit, say 3TB.
> > > > >
> > > > > Couple of questions that I have:
> > > > >
> > > > > 1. Although I've set "index.cache.field.type: soft", GC is not able to
> > > > > reduce heap usage.
> > > > > 2. When I execute that command to clear caches, the indexing stops for
> > > > > a couple of minutes. Why is that?
> > > > > 3. I don't search for documents on all indices. I only use the latest
> > > > > 7 indices or so. Is there a way that I can disable caching for older
> > > > > indices?
> > > > >
> > > > > Clearing caches decreases search performance but I can live with it.
> > > > > Is there a way I can optimize RAM usage by compromising search speed? I
> > > > > just want to limit the size of cache ES uses.
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Abhijeet Rastogi (shadyabhi)
> > > > > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > >
> > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > .
> > >
> > > --
> > > Regards,
> > > Abhijeet Rastogi (shadyabhi)
> > > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> Regards,
> Abhijeet Rastogi (shadyabhi)
> [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)
#### Post date: [April 13, 2013, 5:22am UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528/7 "2013-04-13T05:22:35Z")
No. I can understand why you said that but I haven't done any configuration
for that.
On Fri, Apr 12, 2013 at 10:57 PM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote:
> Are you using warmers?
>
> On Thursday, April 11, 2013 11:28:43 AM UTC-4, Abhijeet Rastogi wrote:
>
> > HI Igor,
> >
> > So, I tested that only if I cleared the latest cache, I got issues of
> > indexing being paused or very slow. used hot\_threads API and it gave me
> > this.
> >
> > 87.8% (878.2ms out of 1s) cpu usage by thread
> > 'elasticsearch[index12][cache]**[T#45]'
> > 87.5% (874.5ms out of 1s) cpu usage by thread
> > 'elasticsearch[index12][cache]**[T#38]'
> > 86.8% (867.7ms out of 1s) cpu usage by thread
> > 'elasticsearch[index12][cache]**[T#44]'
> > 86.3% (863ms out of 1s) cpu usage by thread
> > 'elasticsearch[index12][cache]**[T#46]'
> > 9.8% (97.9ms out of 1s) cpu usage by thread
> > 'elasticsearch[index12][bulk][_T#1373]'
> > 6.1% (61.4ms out of 1s) cpu usage by thread 'elasticsearch[index12][
> > refresh][T#15]'
> > 5.9% (58.8ms out of 1s) cpu usage by thread 'elasticsearch[index12][
> > \*refresh][T#10]'
> > 4.8% (48.2ms out of 1s) cpu usage by thread
> > 'elasticsearch[index12][bulk][**T#1375]'
> > 0.5% (4.9ms out of 1s) cpu usage by thread 'elasticsearch[index12][**
> > scheduler][T#1]'
> > 0.4% (3.9ms out of 1s) cpu usage by thread
> > 'elasticsearch[index12][merge]__[T#6]'
> > 94.0% (939.9ms out of 1s) cpu usage by thread
> > 'elasticsearch[index11][cache]__[T#22]'
> > 92.3% (922.8ms out of 1s) cpu usage by thread
> > 'elasticsearch[index11][cache]__[T#23]'
> > 36.8% (367.5ms out of 1s) cpu usage by thread
> > 'elasticsearch[index11][cache]_\*[T#24]'
> > 7.1% (71.3ms out of 1s) cpu usage by thread 'elasticsearch[index11][\*
> > _refresh][T#7]'
> > 7.0% (70.3ms out of 1s) cpu usage by thread 'elasticsearch[index11][_
> > \*refresh][T#6]'
> > 4.1% (40.8ms out of 1s) cpu usage by thread
> > 'elasticsearch[index11][bulk][\*\*T#1400]'
> > 3.5% (35.2ms out of 1s) cpu usage by thread
> > 'elasticsearch[index11][bulk][\*\*T#1421]'
> > 3.4% (33.9ms out of 1s) cpu usage by thread
> > 'elasticsearch[index11][bulk][\*\*T#1418]'
> > 2.9% (29.4ms out of 1s) cpu usage by thread
> > 'elasticsearch[index11][bulk][\*\*T#1407]'
> > 2.9% (28.5ms out of 1s) cpu usage by thread
> > 'elasticsearch[index11][bulk][\*\*T#1415]'
> >
> > What I don't understand is, even though there is no search activity, ES
> > just starts building cache right away at that very instant of clearing
> > cache. I can see bulk threads so indexing is going on, just that it's very
> > slow.
> >
> > On Wed, Apr 10, 2013 at 11:12 PM, Igor Motov [imo...@gmail.com](mailto:imo...@gmail.com) wrote:
> >
> > > I would guess indexing pauses are caused by I/O and CPU contention.
> > > There are no locks that I am aware of that would prevent indexing from
> > > going on, but by cleaning cache you suddenly force Elasticsearch to reload
> > > Gigabytes of data from disk and rebuild all these caches in memory and it
> > > takes toll on all other processes.
> > >
> > > In order to perform a query that uses a field cache elasticsearch needs
> > > to load entire field cache into memory. So, if you end up with several
> > > queries each trying to load field cache for their set of fields, you might
> > > create a situation when queries are fighting each other over limited memory
> > > evicting each other's cache and trying to load it over and over again.
> > >
> > > Soft cache is cleaned only when it's not used and there is GC pressure.
> > > It's possible that you never achieve these condition. I also suspect
> > > that eviction counter might not be reporting evictions correctly in case of
> > > soft cache.
> > >
> > > On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:
> > >
> > > > Hi Igor,
> > > >
> > > > Thanks for the reply. Can you explain the pauses in indexing when I
> > > > clear cache?
> > > >
> > > > Can you elaborate the part you said about soft cache, I didn't get what
> > > > you meant. Also, why isn't GC clearing the cache even though it's soft?
> > > >
> > > > On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov [imo...@gmail.com](mailto:imo...@gmail.com) wrote:
> > > >
> > > > > If you are not searching older indices you can just close them, which
> > > > > will release cache for these indices. Alternatively, you can clear cache
> > > > > only for indices that you no longer use. Cleaning cache on all indices
> > > > > makes little sense because under heavy search traffic elasticsearch will
> > > > > rebuild all these caches right away causing very sluggish performance in
> > > > > the process. And, by the way, using soft cache is like playing
> > > > > russian roulette - you basically just betting on not having a diverse
> > > > > enough set of queries that will load all values into the cache at the same
> > > > > time and bring your server down.
> > > > >
> > > > > On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > In the beginning itself, I would like to say that the problem can be
> > > > > > solved by adding more nodes or increasing RAM but I can't do that right
> > > > > > now.
> > > > > >
> > > > > > My problem is, every two or three days, I've to execute "curl -XPOST '
> > > > > > [http://localhost:9200/\_cache/\*\*\*\*\*\*clear](http://localhost:9200/_cache/ ****** clear)[http://localhost:9200/\_cache/clear](http://localhost:9200/_cache/clear)'"
> > > > > > to decrease the heap usage. This starts happening after the indices size
> > > > > > grow a certain limit, say 3TB.
> > > > > >
> > > > > > Couple of questions that I have:
> > > > > >
> > > > > > 1. Although I've set "index.cache.field.type: soft", GC is not able
> > > > > > to reduce heap usage.
> > > > > > 2. When I execute that command to clear caches, the indexing stops
> > > > > > for a couple of minutes. Why is that?
> > > > > > 3. I don't search for documents on all indices. I only use the latest
> > > > > > 7 indices or so. Is there a way that I can disable caching for older
> > > > > > indices?
> > > > > >
> > > > > > Clearing caches decreases search performance but I can live with it.
> > > > > > Is there a way I can optimize RAM usage by compromising search speed? I
> > > > > > just want to limit the size of cache ES uses.
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Abhijeet Rastogi (shadyabhi)
> > > > > > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to elasticsearc...@\*\*googlegroups.\*\*com.
> > > > >
> > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > .
> > > >
> > > > --
> > > > Regards,
> > > > Abhijeet Rastogi (shadyabhi)
> > > > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > .
> >
> > --
> > Regards,
> > Abhijeet Rastogi (shadyabhi)
> > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
--
Regards,
Abhijeet Rastogi (shadyabhi)
[http://blog.abhijeetr.com](http://blog.abhijeetr.com)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [April 14, 2013, 4:07pm UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528/8 "2013-04-14T16:07:11Z")
Could you post a complete output of hot\_threads then?
On Saturday, April 13, 2013 1:22:35 AM UTC-4, Abhijeet Rastogi wrote:
> No. I can understand why you said that but I haven't done any
> configuration for that.
>
> On Fri, Apr 12, 2013 at 10:57 PM, Igor Motov \<[imo...@gmail.com](mailto:imo...@gmail.com)\
>
> > wrote:
>
> > Are you using warmers?
> >
> > On Thursday, April 11, 2013 11:28:43 AM UTC-4, Abhijeet Rastogi wrote:
> >
> > > HI Igor,
> > >
> > > So, I tested that only if I cleared the latest cache, I got issues of
> > > indexing being paused or very slow. used hot\_threads API and it gave me
> > > this.
> > >
> > > 87.8% (878.2ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index12][cache]**[T#45]'
> > > 87.5% (874.5ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index12][cache]**[T#38]'
> > > 86.8% (867.7ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index12][cache]**[T#44]'
> > > 86.3% (863ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index12][cache]**[T#46]'
> > > 9.8% (97.9ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index12][bulk][_T#1373]'
> > > 6.1% (61.4ms out of 1s) cpu usage by thread 'elasticsearch[index12][
> > > refresh][T#15]'
> > > 5.9% (58.8ms out of 1s) cpu usage by thread 'elasticsearch[index12][
> > > refresh][T#10]'
> > > 4.8% (48.2ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index12][bulk][T#1375]'
> > > 0.5% (4.9ms out of 1s) cpu usage by thread 'elasticsearch[index12][
> > > \*scheduler][T#1]'
> > > 0.4% (3.9ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index12][merge][T#6]'
> > > 94.0% (939.9ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index11][cache][T#22]'
> > > 92.3% (922.8ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index11][cache][T#23]'
> > > 36.8% (367.5ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index11][cache]_\*[T#24]'
> > > 7.1% (71.3ms out of 1s) cpu usage by thread 'elasticsearch[index11][
> > > \*\*refresh][T#7]'
> > > 7.0% (70.3ms out of 1s) cpu usage by thread 'elasticsearch[index11][
> > > \*\*refresh][T#6]'
> > > 4.1% (40.8ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index11][bulk][\*\*T#1400]'
> > > 3.5% (35.2ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index11][bulk][\*\*T#1421]'
> > > 3.4% (33.9ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index11][bulk][\*\*T#1418]'
> > > 2.9% (29.4ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index11][bulk][\*\*T#1407]'
> > > 2.9% (28.5ms out of 1s) cpu usage by thread
> > > 'elasticsearch[index11][bulk][\*\*T#1415]'
> > >
> > > What I don't understand is, even though there is no search activity, ES
> > > just starts building cache right away at that very instant of clearing
> > > cache. I can see bulk threads so indexing is going on, just that it's very
> > > slow.
> > >
> > > On Wed, Apr 10, 2013 at 11:12 PM, Igor Motov [imo...@gmail.com](mailto:imo...@gmail.com) wrote:
> > >
> > > > I would guess indexing pauses are caused by I/O and CPU contention.
> > > > There are no locks that I am aware of that would prevent indexing from
> > > > going on, but by cleaning cache you suddenly force Elasticsearch to reload
> > > > Gigabytes of data from disk and rebuild all these caches in memory and it
> > > > takes toll on all other processes.
> > > >
> > > > In order to perform a query that uses a field cache elasticsearch needs
> > > > to load entire field cache into memory. So, if you end up with several
> > > > queries each trying to load field cache for their set of fields, you might
> > > > create a situation when queries are fighting each other over limited memory
> > > > evicting each other's cache and trying to load it over and over again.
> > > >
> > > > Soft cache is cleaned only when it's not used and there is GC pressure.
> > > > It's possible that you never achieve these condition. I also suspect
> > > > that eviction counter might not be reporting evictions correctly in case of
> > > > soft cache.
> > > >
> > > > On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:
> > > >
> > > > > Hi Igor,
> > > > >
> > > > > Thanks for the reply. Can you explain the pauses in indexing when I
> > > > > clear cache?
> > > > >
> > > > > Can you elaborate the part you said about soft cache, I didn't get
> > > > > what you meant. Also, why isn't GC clearing the cache even though it's soft?
> > > > >
> > > > > On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov [imo...@gmail.com](mailto:imo...@gmail.com) wrote:
> > > > >
> > > > > > If you are not searching older indices you can just close them, which
> > > > > > will release cache for these indices. Alternatively, you can clear cache
> > > > > > only for indices that you no longer use. Cleaning cache on all indices
> > > > > > makes little sense because under heavy search traffic elasticsearch will
> > > > > > rebuild all these caches right away causing very sluggish performance in
> > > > > > the process. And, by the way, using soft cache is like playing
> > > > > > russian roulette - you basically just betting on not having a diverse
> > > > > > enough set of queries that will load all values into the cache at the same
> > > > > > time and bring your server down.
> > > > > >
> > > > > > On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > In the beginning itself, I would like to say that the problem can be
> > > > > > > solved by adding more nodes or increasing RAM but I can't do that right
> > > > > > > now.
> > > > > > >
> > > > > > > My problem is, every two or three days, I've to execute "curl -XPOST
> > > > > > > '[http://localhost:9200/\_cache/\*\*\*\*\*\*clear](http://localhost:9200/_cache/ ****** clear)[http://localhost:9200/\_cache/clear](http://localhost:9200/_cache/clear)'"
> > > > > > > to decrease the heap usage. This starts happening after the indices size
> > > > > > > grow a certain limit, say 3TB.
> > > > > > >
> > > > > > > Couple of questions that I have:
> > > > > > >
> > > > > > > 1. Although I've set "index.cache.field.type: soft", GC is not able
> > > > > > > to reduce heap usage.
> > > > > > > 2. When I execute that command to clear caches, the indexing stops
> > > > > > > for a couple of minutes. Why is that?
> > > > > > > 3. I don't search for documents on all indices. I only use the
> > > > > > > latest 7 indices or so. Is there a way that I can disable caching for older
> > > > > > > indices?
> > > > > > >
> > > > > > > Clearing caches decreases search performance but I can live with it.
> > > > > > > Is there a way I can optimize RAM usage by compromising search speed? I
> > > > > > > just want to limit the size of cache ES uses.
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Abhijeet Rastogi (shadyabhi)
> > > > > > > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com.
> > > > > >
> > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > > > .
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Abhijeet Rastogi (shadyabhi)
> > > > > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> > > > .
> > >
> > > --
> > > Regards,
> > > Abhijeet Rastogi (shadyabhi)
> > > [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
>
> --
> Regards,
> Abhijeet Rastogi (shadyabhi)
> [http://blog.abhijeetr.com](http://blog.abhijeetr.com)
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:41am UTC](https://discuss.elastic.co/t/caches-and-garbage-collection/11528/9 "2017-07-06T02:41:13Z")