# Bad CPU: when is field cache invalidated? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 2, 2012, 8:17am UTC](https://discuss.elastic.co/t/bad-cpu-when-is-field-cache-invalidated/9209 "2012-10-02T08:17:56Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [October 2, 2012, 8:17am UTC](https://discuss.elastic.co/t/bad-cpu-when-is-field-cache-invalidated/9209/1 "2012-10-02T08:17:56Z")
We are experiencing bad gaps in performance, where the ES machine stops responding for half a minute, consuming all CPUs completely. We narrowed the problem down to field caches. There seems to be no problem with free RAM, but despite that, the field cache sometimes gets invalidated (bigdesk suddenly reports 0B field cache). This leads to field cache loading -- several GB -- which takes a lot of time and is the cause of the GC freeze. Question: under what conditions is the field cache invalidated? Is there any way to invalidate only a part of it (one segment?) instead of this mega-freeze + sudden loading from zero to several GB? Is there a way to pre-load the field cache as soon as possible, and not wait for user queries? Would setting "soft" cache type help? Thank you. -- ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 2, 2012, 6:30pm UTC](https://discuss.elastic.co/t/bad-cpu-when-is-field-cache-invalidated/9209/2 "2012-10-02T18:30:59Z")
As you index data, the field data needs to be reloaded for the "new data" being indexed, and sometimes (due to merging of old+new data) it need to also be loaded for part of the "old" data. When you restart a node, obviously it gets "invalidated". The way to tackle that is solved by using warmers that will be available in upcoming 0.20 version, which will come out probably next week. On Oct 2, 2012, at 4:17 AM, Crwe [tester.testerus@gmail.com](mailto:tester.testerus@gmail.com) wrote: > We are experiencing bad gaps in performance, where the ES machine > stops responding for half a minute, consuming all CPUs completely. > > We narrowed the problem down to field caches. There seems to be no > problem with free RAM, but despite that, the field cache sometimes > gets invalidated (bigdesk suddenly reports 0B field cache). This leads > to field cache loading -- several GB -- which takes a lot of time and > is the cause of the GC freeze. > > Question: under what conditions is the field cache invalidated? > > Is there any way to invalidate only a part of it (one segment?) > instead of this mega-freeze + sudden loading from zero to several GB? > Is there a way to pre-load the field cache as soon as possible, and > not wait for user queries? Would setting "soft" cache type help? > > Thank you. > > -- -- ---
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [October 3, 2012, 4:44pm UTC](https://discuss.elastic.co/t/bad-cpu-when-is-field-cache-invalidated/9209/3 "2012-10-03T16:44:22Z")
On Oct 2, 8:31 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > As you index data, the field data needs to be reloaded for the "new data" being indexed, and sometimes (due to merging of old+new data) it need to also be loaded for part of the "old" data. What we are experiencing is a sharp drop to 0 bytes (field cache as reported by bigdesk), not gradual resizing as parts of "new data" or "old data" get modified. Perhaps the cause could be some mega-segment merge? > When you restart a node, obviously it gets "invalidated". The way to tackle that is solved by using warmers that will be available in upcoming 0.20 version, which will come out probably next week. That sounds good. Looking forward to 0.20 then. > On Oct 2, 2012, at 4:17 AM, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) wrote: > > > We are experiencing bad gaps in performance, where the ES machine > > stops responding for half a minute, consuming all CPUs completely. > > > We narrowed the problem down to field caches. There seems to be no > > problem with free RAM, but despite that, the field cache sometimes > > gets invalidated (bigdesk suddenly reports 0B field cache). This leads > > to field cache loading -- several GB -- which takes a lot of time and > > is the cause of the GC freeze. > > > Question: under what conditions is the field cache invalidated? > > > Is there any way to invalidate only a part of it (one segment?) > > instead of this mega-freeze + sudden loading from zero to several GB? > > Is there a way to pre-load the field cache as soon as possible, and > > not wait for user queries? Would setting "soft" cache type help? > > > Thank you. > > > -- -- ---
**Author:** ![Crwe](https://avatars.discourse-cdn.com/v4/letter/c/898d66/32.png) [@Crwe](https://discuss.elastic.co/u/Crwe)\ **Post date:** [October 3, 2012, 4:48pm UTC](https://discuss.elastic.co/t/bad-cpu-when-is-field-cache-invalidated/9209/4 "2012-10-03T16:48:35Z")
I should add that we are permanently indexing new items, sometimes overwriting old items with the same id in the process. So this is a different scenario to the "append-only log indexing" that I often see mentioned here. On Oct 3, 6:44 pm, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) wrote: > On Oct 2, 8:31 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > As you index data, the field data needs to be reloaded for the "new data" being indexed, and sometimes (due to merging of old+new data) it need to also be loaded for part of the "old" data. > > What we are experiencing is a sharp drop to 0 bytes (field cache as > reported by bigdesk), not gradual resizing as parts of "new data" or > "old data" get modified. Perhaps the cause could be some mega-segment > merge? > > > When you restart a node, obviously it gets "invalidated". The way to tackle that is solved by using warmers that will be available in upcoming 0.20 version, which will come out probably next week. > > That sounds good. Looking forward to 0.20 then. > > > On Oct 2, 2012, at 4:17 AM, Crwe [tester.teste...@gmail.com](mailto:tester.teste...@gmail.com) wrote: > > > > We are experiencing bad gaps in performance, where the ES machine > > > stops responding for half a minute, consuming all CPUs completely. > > > > We narrowed the problem down to field caches. There seems to be no > > > problem with free RAM, but despite that, the field cache sometimes > > > gets invalidated (bigdesk suddenly reports 0B field cache). This leads > > > to field cache loading -- several GB -- which takes a lot of time and > > > is the cause of the GC freeze. > > > > Question: under what conditions is the field cache invalidated? > > > > Is there any way to invalidate only a part of it (one segment?) > > > instead of this mega-freeze + sudden loading from zero to several GB? > > > Is there a way to pre-load the field cache as soon as possible, and > > > not wait for user queries? Would setting "soft" cache type help? > > > > Thank you. > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:10am UTC](https://discuss.elastic.co/t/bad-cpu-when-is-field-cache-invalidated/9209/5 "2017-07-06T03:10:18Z")
Closed on July 6, 2017, 3:10am UTC