Caches and Garbage collection

Hi,

In the beginning itself, I would like to say that the problem can be solved
by adding more nodes or increasing RAM but I can't do that right now.

My problem is, every two or three days, I've to execute "curl -XPOST '
http://localhost:9200/_cache/clear'" to decrease the heap usage. This
starts happening after the indices size grow a certain limit, say 3TB.

Couple of questions that I have:

  1. Although I've set "index.cache.field.type: soft", GC is not able to
    reduce heap usage.
  2. When I execute that command to clear caches, the indexing stops for a
    couple of minutes. Why is that?
  3. I don't search for documents on all indices. I only use the latest 7
    indices or so. Is there a way that I can disable caching for older indices?

Clearing caches decreases search performance but I can live with it. Is
there a way I can optimize RAM usage by compromising search speed? I just
want to limit the size of cache ES uses.

--
Regards,
Abhijeet Rastogi (shadyabhi)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

If you are not searching older indices you can just close them, which will
release cache for these indices. Alternatively, you can clear cache only
for indices that you no longer use. Cleaning cache on all indices makes
little sense because under heavy search traffic elasticsearch will rebuild
all these caches right away causing very sluggish performance in the
process. And, by the way, using soft cache is like playing russian roulette

  • you basically just betting on not having a diverse enough set of queries
    that will load all values into the cache at the same time and bring your
    server down.

On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:

Hi,

In the beginning itself, I would like to say that the problem can be
solved by adding more nodes or increasing RAM but I can't do that right
now.

My problem is, every two or three days, I've to execute "curl -XPOST '
http://localhost:9200/_cache/clear'" to decrease the heap usage. This
starts happening after the indices size grow a certain limit, say 3TB.

Couple of questions that I have:

  1. Although I've set "index.cache.field.type: soft", GC is not able to
    reduce heap usage.
  2. When I execute that command to clear caches, the indexing stops for a
    couple of minutes. Why is that?
  3. I don't search for documents on all indices. I only use the latest 7
    indices or so. Is there a way that I can disable caching for older indices?

Clearing caches decreases search performance but I can live with it. Is
there a way I can optimize RAM usage by compromising search speed? I just
want to limit the size of cache ES uses.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Igor,

Thanks for the reply. Can you explain the pauses in indexing when I clear
cache?

Can you elaborate the part you said about soft cache, I didn't get what you
meant. Also, why isn't GC clearing the cache even though it's soft?

On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov imotov@gmail.com wrote:

If you are not searching older indices you can just close them, which will
release cache for these indices. Alternatively, you can clear cache only
for indices that you no longer use. Cleaning cache on all indices makes
little sense because under heavy search traffic elasticsearch will rebuild
all these caches right away causing very sluggish performance in the
process. And, by the way, using soft cache is like playing russian roulette

  • you basically just betting on not having a diverse enough set of queries
    that will load all values into the cache at the same time and bring your
    server down.

On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:

Hi,

In the beginning itself, I would like to say that the problem can be
solved by adding more nodes or increasing RAM but I can't do that right
now.

My problem is, every two or three days, I've to execute "curl -XPOST '
http://localhost:9200/_cache/**clear http://localhost:9200/_cache/clear'"
to decrease the heap usage. This starts happening after the indices size
grow a certain limit, say 3TB.

Couple of questions that I have:

  1. Although I've set "index.cache.field.type: soft", GC is not able to
    reduce heap usage.
  2. When I execute that command to clear caches, the indexing stops for a
    couple of minutes. Why is that?
  3. I don't search for documents on all indices. I only use the latest 7
    indices or so. Is there a way that I can disable caching for older indices?

Clearing caches decreases search performance but I can live with it. Is
there a way I can optimize RAM usage by compromising search speed? I just
want to limit the size of cache ES uses.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Abhijeet Rastogi (shadyabhi)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I would guess indexing pauses are caused by I/O and CPU contention. There
are no locks that I am aware of that would prevent indexing from going on,
but by cleaning cache you suddenly force Elasticsearch to reload Gigabytes
of data from disk and rebuild all these caches in memory and it takes toll
on all other processes.

In order to perform a query that uses a field cache elasticsearch needs to
load entire field cache into memory. So, if you end up with several queries
each trying to load field cache for their set of fields, you might create a
situation when queries are fighting each other over limited memory evicting
each other's cache and trying to load it over and over again.

Soft cache is cleaned only when it's not used and there is GC pressure.
It's possible that you never achieve these condition. I also suspect
that eviction counter might not be reporting evictions correctly in case of
soft cache.

On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:

Hi Igor,

Thanks for the reply. Can you explain the pauses in indexing when I clear
cache?

Can you elaborate the part you said about soft cache, I didn't get what
you meant. Also, why isn't GC clearing the cache even though it's soft?

On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov <imo...@gmail.com<javascript:>

wrote:

If you are not searching older indices you can just close them, which
will release cache for these indices. Alternatively, you can clear cache
only for indices that you no longer use. Cleaning cache on all indices
makes little sense because under heavy search traffic elasticsearch will
rebuild all these caches right away causing very sluggish performance in
the process. And, by the way, using soft cache is like playing
russian roulette - you basically just betting on not having a diverse
enough set of queries that will load all values into the cache at the same
time and bring your server down.

On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:

Hi,

In the beginning itself, I would like to say that the problem can be
solved by adding more nodes or increasing RAM but I can't do that right
now.

My problem is, every two or three days, I've to execute "curl -XPOST '
http://localhost:9200/_cache/**clearhttp://localhost:9200/_cache/clear'"
to decrease the heap usage. This starts happening after the indices size
grow a certain limit, say 3TB.

Couple of questions that I have:

  1. Although I've set "index.cache.field.type: soft", GC is not able to
    reduce heap usage.
  2. When I execute that command to clear caches, the indexing stops for a
    couple of minutes. Why is that?
  3. I don't search for documents on all indices. I only use the latest 7
    indices or so. Is there a way that I can disable caching for older indices?

Clearing caches decreases search performance but I can live with it. Is
there a way I can optimize RAM usage by compromising search speed? I just
want to limit the size of cache ES uses.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

HI Igor,

So, I tested that only if I cleared the latest cache, I got issues of
indexing being paused or very slow. used hot_threads API and it gave me
this.

87.8% (878.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#45]'
87.5% (874.5ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#38]'
86.8% (867.7ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#44]'
86.3% (863ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#46]'
9.8% (97.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1373]'
6.1% (61.4ms out of 1s) cpu usage by thread
'elasticsearch[index12][refresh][T#15]'
5.9% (58.8ms out of 1s) cpu usage by thread
'elasticsearch[index12][refresh][T#10]'
4.8% (48.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1375]'
0.5% (4.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][scheduler][T#1]'
0.4% (3.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][merge][T#6]'
94.0% (939.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#22]'
92.3% (922.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#23]'
36.8% (367.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#24]'
7.1% (71.3ms out of 1s) cpu usage by thread
'elasticsearch[index11][refresh][T#7]'
7.0% (70.3ms out of 1s) cpu usage by thread
'elasticsearch[index11][refresh][T#6]'
4.1% (40.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1400]'
3.5% (35.2ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1421]'
3.4% (33.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1418]'
2.9% (29.4ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1407]'
2.9% (28.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1415]'

What I don't understand is, even though there is no search activity, ES
just starts building cache right away at that very instant of clearing
cache. I can see bulk threads so indexing is going on, just that it's very
slow.

On Wed, Apr 10, 2013 at 11:12 PM, Igor Motov imotov@gmail.com wrote:

I would guess indexing pauses are caused by I/O and CPU contention. There
are no locks that I am aware of that would prevent indexing from going on,
but by cleaning cache you suddenly force Elasticsearch to reload Gigabytes
of data from disk and rebuild all these caches in memory and it takes toll
on all other processes.

In order to perform a query that uses a field cache elasticsearch needs to
load entire field cache into memory. So, if you end up with several queries
each trying to load field cache for their set of fields, you might create a
situation when queries are fighting each other over limited memory evicting
each other's cache and trying to load it over and over again.

Soft cache is cleaned only when it's not used and there is GC pressure.
It's possible that you never achieve these condition. I also suspect
that eviction counter might not be reporting evictions correctly in case of
soft cache.

On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:

Hi Igor,

Thanks for the reply. Can you explain the pauses in indexing when I clear
cache?

Can you elaborate the part you said about soft cache, I didn't get what
you meant. Also, why isn't GC clearing the cache even though it's soft?

On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov imo...@gmail.com wrote:

If you are not searching older indices you can just close them, which
will release cache for these indices. Alternatively, you can clear cache
only for indices that you no longer use. Cleaning cache on all indices
makes little sense because under heavy search traffic elasticsearch will
rebuild all these caches right away causing very sluggish performance in
the process. And, by the way, using soft cache is like playing
russian roulette - you basically just betting on not having a diverse
enough set of queries that will load all values into the cache at the same
time and bring your server down.

On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:

Hi,

In the beginning itself, I would like to say that the problem can be
solved by adding more nodes or increasing RAM but I can't do that right
now.

My problem is, every two or three days, I've to execute "curl -XPOST '
http://localhost:9200/_cache/****clearhttp://localhost:9200/_cache/clear'"
to decrease the heap usage. This starts happening after the indices size
grow a certain limit, say 3TB.

Couple of questions that I have:

  1. Although I've set "index.cache.field.type: soft", GC is not able to
    reduce heap usage.
  2. When I execute that command to clear caches, the indexing stops for
    a couple of minutes. Why is that?
  3. I don't search for documents on all indices. I only use the latest 7
    indices or so. Is there a way that I can disable caching for older indices?

Clearing caches decreases search performance but I can live with it. Is
there a way I can optimize RAM usage by compromising search speed? I just
want to limit the size of cache ES uses.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Abhijeet Rastogi (shadyabhi)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Are you using warmers?

On Thursday, April 11, 2013 11:28:43 AM UTC-4, Abhijeet Rastogi wrote:

HI Igor,

So, I tested that only if I cleared the latest cache, I got issues of
indexing being paused or very slow. used hot_threads API and it gave me
this.

87.8% (878.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#45]'
87.5% (874.5ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#38]'
86.8% (867.7ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#44]'
86.3% (863ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#46]'
9.8% (97.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1373]'
6.1% (61.4ms out of 1s) cpu usage by thread
'elasticsearch[index12][refresh][T#15]'
5.9% (58.8ms out of 1s) cpu usage by thread
'elasticsearch[index12][refresh][T#10]'
4.8% (48.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1375]'
0.5% (4.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][scheduler][T#1]'
0.4% (3.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][merge][T#6]'
94.0% (939.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#22]'
92.3% (922.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#23]'
36.8% (367.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#24]'
7.1% (71.3ms out of 1s) cpu usage by thread
'elasticsearch[index11][refresh][T#7]'
7.0% (70.3ms out of 1s) cpu usage by thread
'elasticsearch[index11][refresh][T#6]'
4.1% (40.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1400]'
3.5% (35.2ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1421]'
3.4% (33.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1418]'
2.9% (29.4ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1407]'
2.9% (28.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1415]'

What I don't understand is, even though there is no search activity, ES
just starts building cache right away at that very instant of clearing
cache. I can see bulk threads so indexing is going on, just that it's very
slow.

On Wed, Apr 10, 2013 at 11:12 PM, Igor Motov <imo...@gmail.com<javascript:>

wrote:

I would guess indexing pauses are caused by I/O and CPU contention. There
are no locks that I am aware of that would prevent indexing from going on,
but by cleaning cache you suddenly force Elasticsearch to reload Gigabytes
of data from disk and rebuild all these caches in memory and it takes toll
on all other processes.

In order to perform a query that uses a field cache elasticsearch needs
to load entire field cache into memory. So, if you end up with several
queries each trying to load field cache for their set of fields, you might
create a situation when queries are fighting each other over limited memory
evicting each other's cache and trying to load it over and over again.

Soft cache is cleaned only when it's not used and there is GC pressure.
It's possible that you never achieve these condition. I also suspect
that eviction counter might not be reporting evictions correctly in case of
soft cache.

On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:

Hi Igor,

Thanks for the reply. Can you explain the pauses in indexing when I
clear cache?

Can you elaborate the part you said about soft cache, I didn't get what
you meant. Also, why isn't GC clearing the cache even though it's soft?

On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov imo...@gmail.com wrote:

If you are not searching older indices you can just close them, which
will release cache for these indices. Alternatively, you can clear cache
only for indices that you no longer use. Cleaning cache on all indices
makes little sense because under heavy search traffic elasticsearch will
rebuild all these caches right away causing very sluggish performance in
the process. And, by the way, using soft cache is like playing
russian roulette - you basically just betting on not having a diverse
enough set of queries that will load all values into the cache at the same
time and bring your server down.

On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:

Hi,

In the beginning itself, I would like to say that the problem can be
solved by adding more nodes or increasing RAM but I can't do that right
now.

My problem is, every two or three days, I've to execute "curl -XPOST '
http://localhost:9200/_cache/****clearhttp://localhost:9200/_cache/clear'"
to decrease the heap usage. This starts happening after the indices size
grow a certain limit, say 3TB.

Couple of questions that I have:

  1. Although I've set "index.cache.field.type: soft", GC is not able to
    reduce heap usage.
  2. When I execute that command to clear caches, the indexing stops for
    a couple of minutes. Why is that?
  3. I don't search for documents on all indices. I only use the latest
    7 indices or so. Is there a way that I can disable caching for older
    indices?

Clearing caches decreases search performance but I can live with it.
Is there a way I can optimize RAM usage by compromising search speed? I
just want to limit the size of cache ES uses.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

No. I can understand why you said that but I haven't done any configuration
for that.

On Fri, Apr 12, 2013 at 10:57 PM, Igor Motov imotov@gmail.com wrote:

Are you using warmers?

On Thursday, April 11, 2013 11:28:43 AM UTC-4, Abhijeet Rastogi wrote:

HI Igor,

So, I tested that only if I cleared the latest cache, I got issues of
indexing being paused or very slow. used hot_threads API and it gave me
this.

87.8% (878.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#45]'
87.5% (874.5ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache]
[T#38]'
86.8% (867.7ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#44]'
86.3% (863ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache]
[T#46]'
9.8% (97.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1373]'
6.1% (61.4ms out of 1s) cpu usage by thread 'elasticsearch[index12][

refresh][T#15]'
5.9% (58.8ms out of 1s) cpu usage by thread 'elasticsearch[index12][

*refresh][T#10]'
4.8% (48.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1375]'
0.5% (4.9ms out of 1s) cpu usage by thread 'elasticsearch[index12][

scheduler][T#1]'
0.4% (3.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][merge]
[T#6]'
94.0% (939.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache]
[T#22]'
92.3% (922.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache]
[T#23]'
36.8% (367.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache]
*[T#24]'
7.1% (71.3ms out of 1s) cpu usage by thread 'elasticsearch[index11][*
refresh][T#7]'
7.0% (70.3ms out of 1s) cpu usage by thread 'elasticsearch[index11][

*refresh][T#6]'
4.1% (40.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1400]'
3.5% (35.2ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1421]'
3.4% (33.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1418]'
2.9% (29.4ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1407]'
2.9% (28.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1415]'

What I don't understand is, even though there is no search activity, ES
just starts building cache right away at that very instant of clearing
cache. I can see bulk threads so indexing is going on, just that it's very
slow.

On Wed, Apr 10, 2013 at 11:12 PM, Igor Motov imo...@gmail.com wrote:

I would guess indexing pauses are caused by I/O and CPU contention.
There are no locks that I am aware of that would prevent indexing from
going on, but by cleaning cache you suddenly force Elasticsearch to reload
Gigabytes of data from disk and rebuild all these caches in memory and it
takes toll on all other processes.

In order to perform a query that uses a field cache elasticsearch needs
to load entire field cache into memory. So, if you end up with several
queries each trying to load field cache for their set of fields, you might
create a situation when queries are fighting each other over limited memory
evicting each other's cache and trying to load it over and over again.

Soft cache is cleaned only when it's not used and there is GC pressure.
It's possible that you never achieve these condition. I also suspect
that eviction counter might not be reporting evictions correctly in case of
soft cache.

On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:

Hi Igor,

Thanks for the reply. Can you explain the pauses in indexing when I
clear cache?

Can you elaborate the part you said about soft cache, I didn't get what
you meant. Also, why isn't GC clearing the cache even though it's soft?

On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov imo...@gmail.com wrote:

If you are not searching older indices you can just close them, which
will release cache for these indices. Alternatively, you can clear cache
only for indices that you no longer use. Cleaning cache on all indices
makes little sense because under heavy search traffic elasticsearch will
rebuild all these caches right away causing very sluggish performance in
the process. And, by the way, using soft cache is like playing
russian roulette - you basically just betting on not having a diverse
enough set of queries that will load all values into the cache at the same
time and bring your server down.

On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:

Hi,

In the beginning itself, I would like to say that the problem can be
solved by adding more nodes or increasing RAM but I can't do that right
now.

My problem is, every two or three days, I've to execute "curl -XPOST '
http://localhost:9200/_cache/******clearhttp://localhost:9200/_cache/clear'"
to decrease the heap usage. This starts happening after the indices size
grow a certain limit, say 3TB.

Couple of questions that I have:

  1. Although I've set "index.cache.field.type: soft", GC is not able
    to reduce heap usage.
  2. When I execute that command to clear caches, the indexing stops
    for a couple of minutes. Why is that?
  3. I don't search for documents on all indices. I only use the latest
    7 indices or so. Is there a way that I can disable caching for older
    indices?

Clearing caches decreases search performance but I can live with it.
Is there a way I can optimize RAM usage by compromising search speed? I
just want to limit the size of cache ES uses.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.**com.

For more options, visit https://groups.google.com/groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Abhijeet Rastogi (shadyabhi)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Could you post a complete output of hot_threads then?

On Saturday, April 13, 2013 1:22:35 AM UTC-4, Abhijeet Rastogi wrote:

No. I can understand why you said that but I haven't done any
configuration for that.

On Fri, Apr 12, 2013 at 10:57 PM, Igor Motov <imo...@gmail.com<javascript:>

wrote:

Are you using warmers?

On Thursday, April 11, 2013 11:28:43 AM UTC-4, Abhijeet Rastogi wrote:

HI Igor,

So, I tested that only if I cleared the latest cache, I got issues of
indexing being paused or very slow. used hot_threads API and it gave me
this.

87.8% (878.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#45]'
87.5% (874.5ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache]
[T#38]'
86.8% (867.7ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#44]'
86.3% (863ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache]
[T#46]'
9.8% (97.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1373]'
6.1% (61.4ms out of 1s) cpu usage by thread 'elasticsearch[index12][
refresh][T#15]'
5.9% (58.8ms out of 1s) cpu usage by thread 'elasticsearch[index12][
refresh][T#10]'
4.8% (48.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1375]'
0.5% (4.9ms out of 1s) cpu usage by thread 'elasticsearch[index12][

*scheduler][T#1]'
0.4% (3.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][merge]
[T#6]'
94.0% (939.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache]
[T#22]'
92.3% (922.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache]
[T#23]'
36.8% (367.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache]
*[T#24]'
7.1% (71.3ms out of 1s) cpu usage by thread 'elasticsearch[index11][
**refresh][T#7]'
7.0% (70.3ms out of 1s) cpu usage by thread 'elasticsearch[index11][
**refresh][T#6]'
4.1% (40.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1400]'
3.5% (35.2ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1421]'
3.4% (33.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1418]'
2.9% (29.4ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1407]'
2.9% (28.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][**T#1415]'

What I don't understand is, even though there is no search activity, ES
just starts building cache right away at that very instant of clearing
cache. I can see bulk threads so indexing is going on, just that it's very
slow.

On Wed, Apr 10, 2013 at 11:12 PM, Igor Motov imo...@gmail.com wrote:

I would guess indexing pauses are caused by I/O and CPU contention.
There are no locks that I am aware of that would prevent indexing from
going on, but by cleaning cache you suddenly force Elasticsearch to reload
Gigabytes of data from disk and rebuild all these caches in memory and it
takes toll on all other processes.

In order to perform a query that uses a field cache elasticsearch needs
to load entire field cache into memory. So, if you end up with several
queries each trying to load field cache for their set of fields, you might
create a situation when queries are fighting each other over limited memory
evicting each other's cache and trying to load it over and over again.

Soft cache is cleaned only when it's not used and there is GC pressure.
It's possible that you never achieve these condition. I also suspect
that eviction counter might not be reporting evictions correctly in case of
soft cache.

On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:

Hi Igor,

Thanks for the reply. Can you explain the pauses in indexing when I
clear cache?

Can you elaborate the part you said about soft cache, I didn't get
what you meant. Also, why isn't GC clearing the cache even though it's soft?

On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov imo...@gmail.com wrote:

If you are not searching older indices you can just close them, which
will release cache for these indices. Alternatively, you can clear cache
only for indices that you no longer use. Cleaning cache on all indices
makes little sense because under heavy search traffic elasticsearch will
rebuild all these caches right away causing very sluggish performance in
the process. And, by the way, using soft cache is like playing
russian roulette - you basically just betting on not having a diverse
enough set of queries that will load all values into the cache at the same
time and bring your server down.

On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:

Hi,

In the beginning itself, I would like to say that the problem can be
solved by adding more nodes or increasing RAM but I can't do that right
now.

My problem is, every two or three days, I've to execute "curl -XPOST
'http://localhost:9200/_cache/******clearhttp://localhost:9200/_cache/clear'"
to decrease the heap usage. This starts happening after the indices size
grow a certain limit, say 3TB.

Couple of questions that I have:

  1. Although I've set "index.cache.field.type: soft", GC is not able
    to reduce heap usage.
  2. When I execute that command to clear caches, the indexing stops
    for a couple of minutes. Why is that?
  3. I don't search for documents on all indices. I only use the
    latest 7 indices or so. Is there a way that I can disable caching for older
    indices?

Clearing caches decreases search performance but I can live with it.
Is there a way I can optimize RAM usage by compromising search speed? I
just want to limit the size of cache ES uses.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@**googlegroups.**com.

For more options, visit https://groups.google.com/groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.