# Out of memory on cluster with almost no data **URL:** \ **Category:** Elasticsearch\ **Created:** [September 26, 2011, 1:40pm UTC](https://discuss.elastic.co/t/out-of-memory-on-cluster-with-almost-no-data/5449 "2011-09-26T13:40:12Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\ **Post date:** [September 26, 2011, 1:40pm UTC](https://discuss.elastic.co/t/out-of-memory-on-cluster-with-almost-no-data/5449/1 "2011-09-26T13:40:12Z")
Hello Shay. I have an out of memory problem in ES 0.17.5. My cluster contains two data nodes and a lot of short living "test nodes", that doesn't store any data only connect to cluster to do some operations on it. Every "test node" creates its own index with some types and complex mapping and cleans this index before shutdown. After a while my cluster is broken - one of the data nodes throws OOM. It contains then ~100 empty indices. I analyzed heap dump of broken node and almost whole memory is used by thread "elasticsearch[Manikin]clusterService#updateTask-pool-11-thread-1", variable "workQueue" which is of type LinkedBlockingQueue and contains 1798 items of type org.elasticsearch.cluster.service.InternalClusterService$2, with size ~640kB each where updateTask.newState.metaData.indices has size ~491kB and updateTask.newState.routingTable.indicesRouting has size ~102kB. If this update task contains information about all indices and its mapping then its size is ok, but why there are so many update tasks in this queue? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 26, 2011, 5:30pm UTC](https://discuss.elastic.co/t/out-of-memory-on-cluster-with-almost-no-data/5449/2 "2011-09-26T17:30:18Z")
What exactly do you do on each "test node" that connects? Also, is that a client node that connects to the cluster? The thread mentioned is the one responsible for applying cluster wide changes on the master node. For example, when an index is created, it handles the data structures associated with it, and shard placements (not actually moving / placing them, just changing in memory data structures representing them). 2011/9/26 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com) > Hello Shay. > > I have an out of memory problem in ES 0.17.5. > My cluster contains two data nodes and a lot of short living "test nodes", > that doesn't store any data only connect to cluster to do some operations on > it. > Every "test node" creates its own index with some types and complex mapping > and cleans this index before shutdown. > After a while my cluster is broken - one of the data nodes throws OOM. It > contains then ~100 empty indices. > I analyzed heap dump of broken node and almost whole memory is used by > thread "elasticsearch[Manikin]clusterService#updateTask-pool-11-thread-1", > variable "workQueue" which is of type LinkedBlockingQueue and contains 1798 > items of type org.elasticsearch.cluster.service.InternalClusterService$2, > with size ~640kB each > where updateTask.newState.metaData.indices has size ~491kB and > updateTask.newState.routingTable.indicesRouting has size ~102kB. > If this update task contains information about all indices and its mapping > then its size is ok, but why there are so many update tasks in this queue? ---
**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\ **Post date:** [September 27, 2011, 7:31am UTC](https://discuss.elastic.co/t/out-of-memory-on-cluster-with-almost-no-data/5449/3 "2011-09-27T07:31:16Z")
"Test nodes" are of course client nodes. They connect to cluster, create one or two indices with about 10 types. Then index documents there (5000 at most), executes some queries, deletes data in created indices and disconnects. After ~100 similar operations cluster dies with OOM. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2011, 8:49am UTC](https://discuss.elastic.co/t/out-of-memory-on-cluster-with-almost-no-data/5449/4 "2011-09-27T08:49:35Z")
How do they delete the data? Do they delete the content, or the indices? 2011/9/27 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com) > "Test nodes" are of course client nodes. > They connect to cluster, create one or two indices with about 10 types. > Then index documents there (5000 at most), executes some queries, deletes > data in created indices and disconnects. > After ~100 similar operations cluster dies with OOM. ---
**Author:** ![Wojciech\_Durczynski](https://avatars.discourse-cdn.com/v4/letter/w/fbc32d/32.png) [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)\ **Post date:** [September 27, 2011, 9:52am UTC](https://discuss.elastic.co/t/out-of-memory-on-cluster-with-almost-no-data/5449/5 "2011-09-27T09:52:21Z")
They delete content only. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2011, 8:59pm UTC](https://discuss.elastic.co/t/out-of-memory-on-cluster-with-almost-no-data/5449/6 "2011-09-27T20:59:55Z")
can you post the code you use? or a step by step actions you do? 2011/9/27 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com) > They delete content only. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:53am UTC](https://discuss.elastic.co/t/out-of-memory-on-cluster-with-almost-no-data/5449/7 "2017-07-06T03:53:27Z")
Closed on July 6, 2017, 3:53am UTC