HI Igor,
So, I tested that only if I cleared the latest cache, I got issues of
indexing being paused or very slow. used hot_threads API and it gave me
this.
87.8% (878.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#45]'
87.5% (874.5ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#38]'
86.8% (867.7ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#44]'
86.3% (863ms out of 1s) cpu usage by thread
'elasticsearch[index12][cache][T#46]'
9.8% (97.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1373]'
6.1% (61.4ms out of 1s) cpu usage by thread
'elasticsearch[index12][refresh][T#15]'
5.9% (58.8ms out of 1s) cpu usage by thread
'elasticsearch[index12][refresh][T#10]'
4.8% (48.2ms out of 1s) cpu usage by thread
'elasticsearch[index12][bulk][T#1375]'
0.5% (4.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][scheduler][T#1]'
0.4% (3.9ms out of 1s) cpu usage by thread
'elasticsearch[index12][merge][T#6]'
94.0% (939.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#22]'
92.3% (922.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#23]'
36.8% (367.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][cache][T#24]'
7.1% (71.3ms out of 1s) cpu usage by thread
'elasticsearch[index11][refresh][T#7]'
7.0% (70.3ms out of 1s) cpu usage by thread
'elasticsearch[index11][refresh][T#6]'
4.1% (40.8ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1400]'
3.5% (35.2ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1421]'
3.4% (33.9ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1418]'
2.9% (29.4ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1407]'
2.9% (28.5ms out of 1s) cpu usage by thread
'elasticsearch[index11][bulk][T#1415]'
What I don't understand is, even though there is no search activity, ES
just starts building cache right away at that very instant of clearing
cache. I can see bulk threads so indexing is going on, just that it's very
slow.
On Wed, Apr 10, 2013 at 11:12 PM, Igor Motov imotov@gmail.com wrote:
I would guess indexing pauses are caused by I/O and CPU contention. There
are no locks that I am aware of that would prevent indexing from going on,
but by cleaning cache you suddenly force Elasticsearch to reload Gigabytes
of data from disk and rebuild all these caches in memory and it takes toll
on all other processes.
In order to perform a query that uses a field cache elasticsearch needs to
load entire field cache into memory. So, if you end up with several queries
each trying to load field cache for their set of fields, you might create a
situation when queries are fighting each other over limited memory evicting
each other's cache and trying to load it over and over again.
Soft cache is cleaned only when it's not used and there is GC pressure.
It's possible that you never achieve these condition. I also suspect
that eviction counter might not be reporting evictions correctly in case of
soft cache.
On Wednesday, April 10, 2013 1:19:58 PM UTC-4, Abhijeet Rastogi wrote:
Hi Igor,
Thanks for the reply. Can you explain the pauses in indexing when I clear
cache?
Can you elaborate the part you said about soft cache, I didn't get what
you meant. Also, why isn't GC clearing the cache even though it's soft?
On Wed, Apr 10, 2013 at 10:43 PM, Igor Motov imo...@gmail.com wrote:
If you are not searching older indices you can just close them, which
will release cache for these indices. Alternatively, you can clear cache
only for indices that you no longer use. Cleaning cache on all indices
makes little sense because under heavy search traffic elasticsearch will
rebuild all these caches right away causing very sluggish performance in
the process. And, by the way, using soft cache is like playing
russian roulette - you basically just betting on not having a diverse
enough set of queries that will load all values into the cache at the same
time and bring your server down.
On Wednesday, April 10, 2013 8:36:04 AM UTC-4, Abhijeet Rastogi wrote:
Hi,
In the beginning itself, I would like to say that the problem can be
solved by adding more nodes or increasing RAM but I can't do that right
now.
My problem is, every two or three days, I've to execute "curl -XPOST '
http://localhost:9200/_cache/****clearhttp://localhost:9200/_cache/clear'"
to decrease the heap usage. This starts happening after the indices size
grow a certain limit, say 3TB.
Couple of questions that I have:
- Although I've set "index.cache.field.type: soft", GC is not able to
reduce heap usage.
- When I execute that command to clear caches, the indexing stops for
a couple of minutes. Why is that?
- I don't search for documents on all indices. I only use the latest 7
indices or so. Is there a way that I can disable caching for older indices?
Clearing caches decreases search performance but I can live with it. Is
there a way I can optimize RAM usage by compromising search speed? I just
want to limit the size of cache ES uses.
--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.
--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Regards,
Abhijeet Rastogi (shadyabhi)
http://blog.abhijeetr.com
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.