# Elasticsearch becomes unresponsive after updating to 0.90.x **URL:** \ **Category:** Elasticsearch\ **Created:** [June 3, 2013, 10:45am UTC](https://discuss.elastic.co/t/elasticsearch-becomes-unresponsive-after-updating-to-0-90-x/12233 "2013-06-03T10:45:28Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Tasos\_Stathopoulos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tasos_stathopoulos/32/2294_2.png) [@Tasos\_Stathopoulos](https://discuss.elastic.co/u/Tasos_Stathopoulos)\ **Post date:** [June 3, 2013, 10:45am UTC](https://discuss.elastic.co/t/elasticsearch-becomes-unresponsive-after-updating-to-0-90-x/12233/1 "2013-06-03T10:45:28Z")
I am trying to upgrade elasticsearch server to the latest version. We are using thrift in order our application to connect to the elasticsearch. After a while, elasticsearch does not respond at any request. It seems that elasticsearch comes to a deadlock or something but I haven't found a way to reproduce this. The only thing I have is an unresponsive instance. Can you recommend a java tool like jdb that would help me to report this problem properly? I have to mention that there was no problem with 0.19.x versions. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tasos\_Stathopoulos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tasos_stathopoulos/32/2294_2.png) [@Tasos\_Stathopoulos](https://discuss.elastic.co/u/Tasos_Stathopoulos)\ **Post date:** [June 3, 2013, 1:49pm UTC](https://discuss.elastic.co/t/elasticsearch-becomes-unresponsive-after-updating-to-0-90-x/12233/2 "2013-06-03T13:49:44Z")
UPDATE: I found out that the http restfull api responds properly. Whereas thrift does not respond and I can see around 60 connections attached to the thrift port in CLOSE\_WAIT state. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Tasos\_Stathopoulos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tasos_stathopoulos/32/2294_2.png) [@Tasos\_Stathopoulos](https://discuss.elastic.co/u/Tasos_Stathopoulos)\ **Post date:** [June 4, 2013, 1:37pm UTC](https://discuss.elastic.co/t/elasticsearch-becomes-unresponsive-after-updating-to-0-90-x/12233/3 "2013-06-04T13:37:28Z")
[https://lh5.googleusercontent.com/-tUS2JxI0xtE/Ua3sSFpXfeI/AAAAAAAAAE0/ovY3GPsoD60/s1600/heap-memory-0-90-1.png](https://lh5.googleusercontent.com/-tUS2JxI0xtE/Ua3sSFpXfeI/AAAAAAAAAE0/ovY3GPsoD60/s1600/heap-memory-0-90-1.png) [https://lh5.googleusercontent.com/-8RhVQ\_L8Cg8/Ua3rvelmDfI/AAAAAAAAAEs/g9gGcbsupRk/s1600/heap-memory-0-19-12.png](https://lh5.googleusercontent.com/-8RhVQ_L8Cg8/Ua3rvelmDfI/AAAAAAAAAEs/g9gGcbsupRk/s1600/heap-memory-0-19-12.png) I managed to reproduce the problem. It seems that successive index creation and deleting the contents of an index using delete by query with match\_all makes the server to utilize the whole heap memory. After a while the server becomes unresponsive. This happens at versions 0.90.x and 0.19.12 that I have tested. The difference is that in the 0.19.8 version, the garbage collector is applied and the heap is empty. The servers starts to run again without any problem. At the version 0.90.1, although the garbage collector is applied the heap memory hasn't been reduced at all. I don't know if this is related with this issue: [https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/w370mdz3QDA](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/w370mdz3QDA) The script that I used to reproduce the problem is the following: #!/bin/bash while true; do echo "Deleting all indexes" curl -XDELETE '[http://localhost:9200/\_all/?pretty](http://localhost:9200/_all/?pretty)' \> /dev/null 2\>&1 echo "Creating index" curl -XPUT '[http://localhost:9200/test/?pretty](http://localhost:9200/test/?pretty)' -d '{ "index" : true, "settings": { "number\_of\_shards" : 1 }, "_default_": { "include\_in\_all" : true }, "mappings" : { "blog\_post" : { "type" : "object", "\_all" : {"enabled" : true }, "\_source": {"enabled" : true, "compress" : true} } } }' \> /dev/null 2\>&1 for i in {1..10} do echo "Truncating $i times" curl -XDELETE '[http://localhost:9200/test/blog\_post/\_query?pretty](http://localhost:9200/test/blog_post/_query?pretty)' -d '{ "match\_all" : {} }' \> /dev/null 2\>&1 ``` echo "Refreshing $i times" curl -XPOST 'http://localhost:9200/test/_refresh?pretty' > /dev/null 2 ``` > &1 > done done -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Usman\_Masood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/usman_masood/32/2275_2.png) [@Usman\_Masood](https://discuss.elastic.co/u/Usman_Masood)\ **Post date:** [June 20, 2013, 10:42pm UTC](https://discuss.elastic.co/t/elasticsearch-becomes-unresponsive-after-updating-to-0-90-x/12233/4 "2013-06-20T22:42:39Z")
We are experiencing a similar problem. In our case, the nodes are heavy duty with 15GB of memory (8GB mlocked by ES) and are part of a single cluster. Previously we used to observe the heap memory dropping significantly (maybe to 2-3GB) once it got close to 8GB (probably because the GC kicked in). However after upgrading to 0.90, we see that the heap memory keeps on increasing and plateaus around 7-8GB. When that happens CPU utilization suddenly spikes and the GC also starts happening much more frequently. We've tried restarting the cluster but to no effect. On Tuesday, June 4, 2013 6:37:28 AM UTC-7, Tasos Stathopoulos wrote: > [https://lh5.googleusercontent.com/-tUS2JxI0xtE/Ua3sSFpXfeI/AAAAAAAAAE0/ovY3GPsoD60/s1600/heap-memory-0-90-1.png](https://lh5.googleusercontent.com/-tUS2JxI0xtE/Ua3sSFpXfeI/AAAAAAAAAE0/ovY3GPsoD60/s1600/heap-memory-0-90-1.png) > > [https://lh5.googleusercontent.com/-8RhVQ\_L8Cg8/Ua3rvelmDfI/AAAAAAAAAEs/g9gGcbsupRk/s1600/heap-memory-0-19-12.png](https://lh5.googleusercontent.com/-8RhVQ_L8Cg8/Ua3rvelmDfI/AAAAAAAAAEs/g9gGcbsupRk/s1600/heap-memory-0-19-12.png) > I managed to reproduce the problem. It seems that successive index > creation and deleting the contents of an index using delete by query with > match\_all makes the server to utilize the whole heap memory. After a while > the server becomes unresponsive. > This happens at versions 0.90.x and 0.19.12 that I have tested. The > difference is that in the 0.19.8 version, the garbage collector is applied > and the heap is empty. The servers starts to run again without any problem. > > At the version 0.90.1, although the garbage collector is applied the heap > memory hasn't been reduced at all. > > I don't know if this is related with this issue: > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/w370mdz3QDA) > > The script that I used to reproduce the problem is the following: > > #!/bin/bash > > while true; do > > echo "Deleting all indexes" > curl -XDELETE '[http://localhost:9200/\_all/?pretty](http://localhost:9200/_all/?pretty)' \> /dev/null 2\>&1 > > echo "Creating index" > curl -XPUT '[http://localhost:9200/test/?pretty](http://localhost:9200/test/?pretty)' -d '{ > "index" : true, > "settings": { > "number\_of\_shards" : 1 > }, > "_default_": { > "include\_in\_all" : true > }, > "mappings" : { > "blog\_post" : { > "type" : "object", > "\_all" : {"enabled" : true }, > "\_source": {"enabled" : true, "compress" : true} > } > } > }' \> /dev/null 2\>&1 > > for i in {1..10} > do > echo "Truncating $i times" > curl -XDELETE '[http://localhost:9200/test/blog\_post/\_query?pretty](http://localhost:9200/test/blog_post/_query?pretty)' -d > '{ > "match\_all" : {} > }' \> /dev/null 2\>&1 > > ``` > echo "Refreshing $i times" > curl -XPOST 'http://localhost:9200/test/_refresh?pretty' > /dev/null > > ``` > > 2\>&1 > done > > done -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:30am UTC](https://discuss.elastic.co/t/elasticsearch-becomes-unresponsive-after-updating-to-0-90-x/12233/5 "2017-07-06T02:30:06Z")